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FOREWORD 

} The  i5th  Annual  Conference  of  the  Military  Testing  Association  was 

I held  at  the  El  Tropicano  Motor  Hotel,  San  Antonio,  Texas,  28  October  - 

j 2 November  1973. 

j The  papers  presented  at  the  15th  Annual  Conference  of  the  MTA  reflect 

a diversity  of  subject  matter.  These  presentations  from  representatives 
of  both  the  military  and  civilian  community  reflect  the  opinions  of  the 
authors  and  are  not  to  be  construed  as  official  or  any  way  representative  of 
the  views  and  policies  of  the  United  States  Armed  Services  or  the  armed 
services  of  the  foreign  countries  participating  in  this  meeting. 
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- OFFICIAL  PROGRAM  - 


15th  Annual  Conference 
MILITARY  TESTING  ASSOCIATION 


28  October  through  2 November  1973 


Sunday,  October  28 

El  Tropicano  Lobby 
1300  - 1700 

Registration 

Monday,  October  29 

El  Tropicano  Lobby 
0800  - 1600 

Registration 

Hemisfair  Room 
Ground  Floor 

Session  Chairman 

Col  Theodore  B,  Aldrich 

Air  Force  Human  Resources  Laboratory 

0900 

Conference  called  to  order 

0900  - 0915 

Greetings  by  AFHRL  Commander 
Col  Harold  E.  Fischer 
Air  Force  Human  Resources  Laboratory 

0915  - 0945 

Keynote  Address 
Lt  Gen  John  W.  Roberts 
Deputy  Chief  of  Staff  Personnel 
Headquarters,  United  States  Air  Force 

0945  - 1010 

Introductory  Comments 
Col  Theodore  B.  Aldrich 
Air  Force  Human  Resources  Laboratory 

1010  - 1015 

Administrative  Announcements 

1015  - 1030 

Coffee  Break 
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PRSCEDI  .0  P^3E  BUNK-::CT  flU-KD 


Symposium 

EoAly-6ZfLV'Lcz  6CAZzning  ^ofi  mitita/uj  peJi{^oHmancz 
and  bzhaviofi 

Chai rman 

Dr.  Ralph  R.  Canter 

Office  of  the  Assistant  Secretary  of  Defense 
(M&RA) 

Participants 
Maj  James  Taylor 
United  States  Air  Force 
Comdr  James  Murphy 
United  States  Navy 
Lt  Col  Todd  Graham 
Uni  ted  States  Army 
Dr.  D.  Brian  Murphy 
Systems  Development  Corporation 
Dr.  Robert  G.  Smith 
Human  Resources  Research  Organization 

Discussant 

Dr.  Norman  J.  Kerr 

Naval  Technical  Training  Command 


1200  - 1330 


Peveiopimnt  oi  the  kfmed  SeAvtce6  Vocational 
Aptitude  BatteAif 
Dr.  M.  A.  Fischl 

United  States  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 


Coffee  Break 


505  - 1535 


535  - 1550 


550  - 1610 


Session  Chairman 

Capt  James  A.  Hoskins 

Air  Force  Human  Resources  Laboratory 

Wcxo  deveiopmenti)  in  de^eme  ianguage  pro^icienaj 
tei  ting 

Drs.  C.  D.  Leatherman  and  A.  Al-Haik 

United  States  Army,  Defense  Language  Institute 

Presented  by:  Mr.  Sydney  Sako 

Entiited  s ejection  and  clarification  testing  in 
the  U.  S.  A‘tmi/ 

Dr.  Milton  H.  Maier 

United  States  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
Presented  by:  Dr.  M.  A.  Fischl 

The  development  of  Engttsh  Comprehension  Level 
screening  tests  for.  foreign  students 
Mr.  Cortez  Parks 

United  States  Army,  Defense  Languaoe  Institute 


Monterrey  Suite 
Room  909 


Acapulco  Room 
Room  902 


END  OF  MONDAY  SESSIONS 


MTA  President's  welcoming  and  social  hour 


Tuesday,  October  30 


Regi strati  on 


A 
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Hemisfair  Room 
Ground  Floor 

0845  - 0900 
0900  - 0920 


0920  - 0945 


0945  - 1000 


1000  - 1155 


Session  Chairman 
Capt  James  A.  Hoskins 
Air  Force  Human  Resources  Laboratory 

Call  to  order 

An  zxpeAlmental,  muCtcrmcUa  coAzeA  devztopment 
cou/ae  (Jo/i  nexv  mental  itanda/ids  alamen 
Mr.  George  P.  Scharf 

USAF  School  of  Applied  Aerospace  Sciences 

Optimal  utyitization  on- the- job  training  and 

technical  tAolning  school 
Capt  Alan  0.  Dunham 
National  Security  Agency 


Coffee  Break 


Session  Chairman 
Mr.  Manuel  Pina 

Air  Force  Human  Resources  Laboratory 
Symposium 

Tfumilation  o^  Vialnlng  fiCySeoAch  into  tAolning 
action  - a mining  link 

Chai rman 

Dr.  G.  Douglas  Mayo 

Naval  Technical  Training  Command 

Participants 

FA-Orn  the  vieiopolnt  o(,  a training  manageA 

Mr.  Walter  E,  McDowell 

Arn\y  Training  and  Doctrine  Command 

FAom  the  viewpoint  o(i  a tAoi.ning  AeteoAcheA 
Dr.  Norman  J.  Kerr 
Naval  Technical  Training  Command 

Discussants 
Lt  Col  Donald  F.  Mead 
Air  Training  Command 
Dr.  Earl  I.  Jones 

Navy  Personnel  Research  & Development  Center 
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Reaction  Panel 
Comdr  Bruce  Cormack 
Canadian  Armed  Forces 
Mr.  William  B.  Lecznar 
Air  Force  Human  Resources  Laboratory 
Mr.  Ralph  Canter 

Office  of  the  Assistant  Secretary  of  Defense 
(M&RA) 

1155  - 1200 

Administrative  Announcements 

1200  - 1330 

Lunch 

Monterrey  Suite 
Room  909 

1330  - 1500 

Meetinq  of  the  Steering  Committee 

Hemisfair  Room 
Ground  Floor 

Session  Chairman 
Mr.  William  J.  Stacy 
Air  Force  Human  Resources  Laboratory 

1330  - 1340 

Administrative  Announcements 

1340  - 1400 

Te,it  feedback  and  PiainZng  objectives 
LtJG  Carroll  H.  Greene 
United  States  Coast  Guard  Training  Center 

1400  - 1420 

"PROBE"  - An  automated  testing  system  ijot  the 
eval'jdXticn  c{,  abiiitij  and/ox  iviitingnci6  to 
profit  ^xom  a txaintng  situation 
Dr.  David  B.  Vinson 
Assessment  Systems,  Incorporated 

1420  - 1445 

Pxoipective  c/j-ccjf  petty  o^i^icex  Zeadexsliip/ 
management  semnax 
Comdr  G.  C.  Hinson 

United  States  Coast  Guard  Training  Center 

1445  - 1 510 

Txanscendentat  Meditation  (TM) : A tkxeat  ox 
technique  in  ^utuxe  pexsonnei.  evaluation 
Dr.  R.  0.  Waldkoetter 

Consultant,  Education  and  Personnel  Systems 

1510  - 1525 

Coffee  Break 

ix 


1525  - 1545 


Session  Chairman 

Capt  Nicholas  C.  Varney 

Air  Force  Human  Resources  Laboratory 

JmpticatXoni  coM.e.1  instruction  on  mititoAij 
tcs  ting 

Dr.  Ronald  W.  Spangenberg 

Air  Force  Human  Resources  Laboratory 

Presented  by:  Dr.  Roger  J.  Pennell 


1545  - 1600 


imeti-t  o{^  complex  psydiomotor  coordination 
learning  and  pcrf^onmance  during  130  days  o^ 
intensioe  testing 
Dr.  Randall  M.  Chambers 
Georgia  Institute  of  Technology 
Dr.  Rayford  T.  Saucer 
Veterans  Administration  Hospital 


1600  - 1630 


Proficiency  measurement  in  flight  simulators 
Dr.  Edwin  Cohen 

The  Singer  Company,  Simulation  Products  Division 


END  OF  TUESDAY  SESSIONS 


Wednesday,  October  31 

Hemisfair  Room 
Ground  Floor 

Session  Chairman 
Mr.  Andrew  T.  Garza 
Air  Force  Human  Resources  Laboratory 

0830 

Call  to  order 

0830  - 0850 

The  personnel  and  training  evaluation  program: 
A leorking  program  for  improving  the  efficiency 
and  effectiveness  of  fleet  ballistic  inissile 
iceofoons  sys  tem  training~Part  7 - scope  and 
achievemen t 
Lt  Comdr  C.  L.  Walker 
Naval  Guided  Missiles  School 

0850  - 0920 

The  personnel  and  training  evaluation  program: 
A wording  program  for  improving  the  efficiency 
and  effectiveness  of  fleet  balZistic  missile 
weapons  system  training-Part  II  - program 
development 

Mr.  Frank  B.  Braun 
Data  Design  Laboratories 

X 


0920  - 

0935 

Invantcnij  and  ai6U6rmnt  at  the.  linlttd  States 
AAiruj  Wan.  Coltzge. 

Lt  Col  Quay  C.  Snyder 
United  States  Arniy  War  College 

0935  - 

0950 

Subondinatz  natingi:  Wliij  not? 

Drs.  W.  H,  Githens  and  R.  S.  Els  ter 
United  States  Naval  Postgraduate  School 

0950  - 

1005 

Thz  KzzitcA  Stadij-zlzctnonic  tzchniciam  ^ou/i 
yzan  zvaluation  thnzz  tijpzi  o{,  tnatning 

Dr.  Virginia  Zachert 
Medical  College  of  Georgia 

1005  - 

1020 

Coffee  Break 

Session  Chairman 
Mr.  Kenneth  G.  Koym 
Air  Force  Human  Resources  Laboratory 

1020  - 

1035 

Pnoczii  vzuiiS  pnoducX  me.cu>uAZA  in  pzAffOnmancz 
tz!)  ting 

Mr.  William  C.  Osborn 

Human  Resources  Research  Organization 

1035  - 

1050 

Tkz  Se£j(  Evatiuttton  TzcJintquz  (SET!  itadi.zA 
Dr.  John  J.  Holden 

United  States  Army  Ordnance  Center  and  School 

1050  - 

1120 

A modiitan  approach  to  pno^iciznzij  tziting 
Dr.  Robert  W.  Stephenson,  Mr.  Warren  P.  Davis, 
and  Mr.  Harry  I.  Hadley 
American  Institutes  for  Research 

Mrs.  Bertha  H.  Cory 

United  States  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 

1120  - 

1200 

Tkz  gnawing  dzmand  ^on  human  pzn^onmojicz  tziting 
Mr.  J.  E.  Gerber,  Jr. 

United  States  Army  Infantry  School 
Presented  by:  Capt  John  P.  Otjen 

1200  - 

1330 

Lunch 

Hemisfair  Room 
Ground  Floor 

Session  Chairman 
Lt  James  W.  Abellera 
Air  Force  Human  Resources  Laboratory 

xi 


1330  - 1350 


1350  - 1420 


1420  - 1435 


1435  - 1450 


C>viteAlon-''izi<iAznc.zd  pzK{^onma.nc.z  tuting 
■in  combat  a/imi  ifiitti 
Mr.  John  F.  Hayes 
URS/Matrix  Company 

The  kin.  Tonce  uiif^c:  Hen.  knoiclcdgc  o{t,  and 
attitadej)  tcMojid,  the  kiK  Fonce 
Drs.  John  A.  Belt  and  Arthur  B.  Sweney 
Wichita  State  University 

Cadet  Ketention  at  the  Rogai.  hiititantj  College  o{^ 
Canada  ai  a function  o^  pen^onalltij , leadeukip 
pen^onmance,and  biographical  vaniablet 
Lt  Col  G.  J.  Carpenter 
Royal  Military  College  of  Canada 

Motivation  o^  enZLited  neAcnvliti  in  a non-dra^t 
atmoi phene 
Lt  Knight  D.  Cheney 
United  States  Coast  Guard  Reserve 


1450  - 1505 


Encounter  group-i  an  a meam  ior  e^^ecting 
attitudinal  diange 
Dr.  Peter  F.  Newton 
National  Security  Agency 


1505  - 1520 


Coffee  Break 


Session  Chairman 
Mr.  James  M.  Wil bourn 
Air  Force  Human  Resources  Laboratory 


1520  - 1540 


An  evaluation  OjJ  diagnostic  vocabuCarg  testing 
In  kir  Force  technicjxZ  training 
Lt  William  P,  Mockovak 
Air  Force  Human  Resources  Laboratory 


1540  - 1600 


Uni(^ormed  phgsiciayvs  icithout  patients:  A 
squandering  of,  scarce  ■lesounce.s? 

Dr.  Gary  B.  Brumback 

American  Institutes  for  Research 


1600  - 1645 


An  experimental  implementation  o^  computer 
assisted  admissible  probabilitij  testing 
Mr.  W.  L.  Sibley 
RAND  Corporation 


END  OF  WEDNESDAY  SESSIONS 


Monterrey  Sui te 
Room  909 


1730  - 1830 


MTA  President's  social  hour 


Thursday,  November  1 


Hemisfair  Room 
Ground  Floor 


Session  Chairman 
Lt  Barry  P.  McFarland 
Air  Force  Human  Resources  Laboratory 


0830 


Call  to  order 


0830  - 0900 


CHAROSEL:  A compiite.n  p'lcg-raiv  ivfUcli  izlzcti 
qual.ita-tlvc  p^izcUctc^s  qa.cUilta.tivz  zxitzxicn 
pxzdiction  pfiobtzm 
Dr.  William  J.  Moonan 

Navy  Personnel  Research  and  Development  Center 


0900  - 0915 


A (^acton.  anatijtic  approach  to  tliz  olltc^xion 
paobtzm 

Dr.  Roger  Pennell 

Air  Force  Human  Resources  Laboratory 


0915  - 1005 


Applicationi  oi  p-rzdictc'i  o\dz.ung  and  izlectlcn 
bij  Saijz,iian-dcci.sloy\  tzclinlquc 
Mr.  S.  E.  Bowser 

Navy  Personnel  Research  and  Development  Center 


1005  - 1020 


Coffee  Break 


Session  Chairman 
Mr.  William  J.  Phalen 
Air  Force  Human  Resources  Laboratory 


1020  - 1045 


Hotv  admisiibCz  pxcbab  ititij  tz^sting  a^/{zcti  and 
ti  a^^zcted  by  thz  largza  sgitzm  incziTtivzi 
Dr.  Thomas  A.  Brown 
RAND  Corporation 


r 
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1045  - 1100 


1100  - 1130 


G/ujw-cng  dcimndi>  on  human  fie^ouAcu:  A vZeui 
^Kom  induiitAif 

Dr.  Thomas  C.  Tuttle 

Westinghouse  Behavioral  Safety  Center 

Rav-ieu'  oi  Kin.  Foncz  job  6aii6 faction  and  came.n 
dzvzZopmaut  ne^e.andi 
Mr.  R.  Bruce  Gould 

Air  Force  Human  Resources  Laboratory 


1130  - 1145 


A m^tiiodologu  f,on  dzteAmining  job  -iotu^action 
in  the  II,  S,  ^avef 

Dr.  Lawrence  A.  Goldman 

Naval  Occupational  Task  Analysis  Program 


1145  - 1200 


JntM&it  iunvagi,  and  job  piacmani  in  tliz  Anm&d 
SznoiceA 

Dr.  William  J.  Figel 
Science  Research  Associates 


1200  - 1330 


Lunch 


Hemisfair  Room 
Ground  Floor 


Session  Chairman 
Mr.  C.  Amos  Johnson 
Air  Force  Human  Resources  Laboratory 


1330  ~ 1355 


I 1355  - 1420 

! 

I 

I 

I 


Job  dt^cniption  in  the.  Knmed  fonce,i  ai  a ba6c6 
()On  anaiijAii)  and  zvaiaation 
Flottillenadmiral  Guenter  Fiebig 
Armed  Forces  Staff 
German  Federal  Ministry  of  Defense 
Presented  by:  Lt  Col  H.  E.  Seuberlich 

?066ibitU:ieyS  and  timitatiom>  job 
and  job  evaluation  in  the  Anmed  fonees 
Lt  Col  H.  E.  Seuberlich 
Armed  Forces  Staff 
German  Federal  Ministry  of  Defense 


1420  - 1440 


Occupational  anal^-iii  in  the  Ro^al  Aaittnallan 
Ain  Fonce 

Capt  Wayne  S.  Sell  man 
United  States  Air  Force 
Sqn  Ldr  John  W,  K.  Fugill 
Royal  Australian  Air  Force 
Presented  by: 

Sqn  Ldr  William  J.  Watson 

Air  Force  Human  Resources  Laboratory 
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1440  - 1500 


XV 


0845  - 0900 


Administrative  Announcements 


0900  - 0920 


0920  - 0940 


Admini^ tAotlon  mat tiple- choice.  te.^ti  to 

non-^cadcAi  via  tape  ^eco^deA;  CoJi>e  itudieA 
Dr.  Joseph  L.  Boyd,  Jr. 

Educational  Testing  Service 

Racial  di^^eAenccs  in  AFQT,  AQE,  and  WAJS  icoKZA 
o{i  Aifi  Fo-ice  baiic  tAoineu 
Capt  Stephen  B.  Knouse,  Sgt  David  F.  McGrevy, 
and  Sgt  Ronnie  A.  Thompson 
Air  Force  Human  Resources  Laboratory 


0940  - 1005 


Voc6  the  USAF  O^^iceA  BiogAopkical  Jnveiitofu^ 
position  0^  the  AFOQT  InadveAtenthj  mcaiuAe 
aathoAitoAian  peuonalitij? 

Lt  Col  Charles  W.  Haney 

Air  Force  Reserve  Officer  Training  Corps 


1005  - 1020 


Coffee  Break 


Session  Chairman 
Mr.  Larry  T.  Looper 
Air  Force  Human  Resources  Laboratory 


1020  - 1035 


Himan  n.elation,!,  evaluation 

Ms.  A.  M.  Kelleher 

United  States  Coast  Guard  Training  Center 


1035  - 1100 


CentAolized  selection  njitem  f,o^  Ail  Fo-ice 
pilots 

Lt  Col  Melvin  S.  Majesty 

USAF  School  of  Military  Sciences,  Officer 


1100  - 1130 
1130  - 1150 


United  States  Amij  pilot  assessment  centeA  project 
Col  Wallace  F.  Veaudry 
United  States  Army  Infantry  School 

Assessment  cetiteAS:  Validation  o^  theiA  use  i^oa 
selection,  tAaini.ng,  ojirf  coAeeA  development  in  a 
mititoAij  seM'ing 
Dr.  Kay  H.  Smith 

United  States  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences 


xvi 


1150  - 1210 

A p->ioposecl  ipatial  ofUtntationfdiiofiimtation 
(ttiglit  tAdining  concept 
Mr.  Patrick  J.  Dowd 
USAF  School  of  Aerospace  Medicine 

1210  - 1230 

Concluding  Comments 

END  OF  CONFERENCE 
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KEYNOTE  ADDRESS 


Lt  Gen  John  W.  Roberts 
Deputy  Chief  of  Staff  Personnel 
Headquarters,  United  States  Air  Force 

It's  traditional  for  a speaker,  particularly  a keynote 
speaker,  to  open  his  remarks  with  the  statement  that  he's 
pleased  and  gratified  to  be  wherever  he  is.  I can  do  this 
with  a considerable  measure  of  sincerity,  since  being  here 
gives  me  an  opportunity  to  pass  on  some  thoughts  to  a group 
that  holds  the  key  to  better  use  of  human  resources  in  these 
days  of  growing  demands. 

In  the  last  few  davs,  I've  supplemented  my  knowledge 
about  this  audience  with  some  homework,  including  a review 
of  the  minutes  of  your  past  meetings.  Your  achievements,  both 
as  individuals  and  as  an  association,  are  certainly  more  than 
impressive . 

Although  I do  not  speak  to  you  as  a psychologist  or  as 
a measurement  specialist,  I feel  a definite  kinship  with  many 
in  this  group.  I have  worked  very  closely  with  people  in  the 
Human  Resources  Laboratory  for  the  past  three  years.  As 
Director  of  Personnel  Plans  for  the  Air  Force,  and  now  as 
the  Deputy  Chief  of  Staff  for  Personnel,  I know  that  many  of 
the  decisions  we  have  made,  and  will  make,  are  based  directly 
on  the  results  of  your  efforts.  Your  contributions  haye  had 
a significant  impact  on  the  way  we  manage  people;  only  the  most 
naiye  would  deny  that  we've  been  able  to  build  a far  better 
military  establishment  with  you  than  we  could  haye  without  you. 
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For  these  reasons,  then,  I welcome  the  opportunity  to 
pass  on  some  of  my  thoughts  and,  hopefully,  to  offer  some 
challenges  for  the  future.  I'll  start  with  a couple  of 
observations . 

In  reading  over  past  conference  reports,  I find  that 
your  guest  speakers  can  be  broadly  categorized  into  two 
groups.  In  the  first  category  are  the  personnel  researchers, 
the  experts,  people  like  you.  They  understand  the  behavioral 
sciences . They  understand  the  problems  and  the  lead  times 
involved  in  achieving  effective  progress.  As  a result,  they 
usually  tell  an  audience  like  this  to  ride  out  the  changes 
in  administrative  personnel,  to  avoid  a preoccupation  with 
the  operational  demands  of  today's  job,  to  take  the  long 
range  view  in  the  interest  of  excellence  and  real  progress. 

In  the  second  category,  I find  people  like  myself, 
managers,  or  laymen  (in  the  research  vernacular)  trying  to 
cope  with  the  day-to-day  problems  of  procuring  and  managing 
large  numbers  of  people.  While  we  realize  the  futility  of 
trying  to  come  up  with  airtight  solutions,  we  tend  to  ask 
you  for  formulas  and  tools  (what  you  might  call  gimmicks) 
that  will  enable  us  to  tie  up  our  problems  in  nice,  neat 
packages.  And,  of  course,  we're  always  in  a hurry.  We 
become  a little  impatient  when  your  answers  are  not  quick 
enough  or  you  try  to  qualify,  perhaps  justifiably,  the  answers 
you  do  provide. 
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From  experience,  I know  that  the  difference  in 
perspective  represented  in  these  two  points  of  view  can 
put  the  manager  and  his  experts  at  odds.  I would  go  even 
further.  They  can,  and  often  do,  make  effective  communication 
difficult,  if  not  impossible.  The  tragic  thing  is  that  we  have 
never  needed  effective  communication  between  the  researcher 
and  the  manager  more  than  we  do  today.  The  demands  on  our 
ingenuity  to  employ,  to  challenge,  to  get  maximum  value  from 
those  who  elect  to  join  us  have  never  been  greater.  There  is 
too  much  to  know,  too  much  to  comprehend,  too  much  that  needs 
to  get  done,  to  waste  time  with  jargon  and  petty  bickering. 

At  this  point,  let  me  share  a few  personal  reflections 
I believe  to  be  crucial  to  understanding  the  nature  and  scope 
of  the  challenges  that  confront  us. 

I think  that  any  assessment  of  our  current  situation  has  to 
begin  with  the  basic  premiise  that  as  a nation  we  are  passing 
through  some  very  tough  times,  perhaps  the  toughest  in  our 
history.  It  is  a period  characterized  on  one  hand  by 
questioning  and  uncertainty  and  on  the  other  by  an  awakened 
social  awareness  in  almost  every  sector  of  society.  Institutions, 
as  well  as  basic  values  and  standards,  are  being  challenged. 

Change  and  turbulence  have  become  the  rule  rather  than  the 
exception . 

You  can  attribute  what  you  read  and  hear  to  a lot  of 
things.  But  whether  you  blame  the  current  climate  on  change, 
disenchantment  with  the  "establishment,"  the  war,  the  volunteer 
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effort,  or  some  other  factor,  the  fact  remains  that  today's 
environment  has  some  profound  implications  for  the  armed 
forces,  especially  when  it  comes  to  manpower. 

We  can  talk  about  this  situation  in  terms  of  tight 
money.  We  can  point  to  force  reductions,  or  shortfalls  in 
military  strength.  What  it  all  means,  however,  is  that  the 
military,  as  an  institution,  must  face  up  to  the  challenge 
of  having  to  manage  its  human  assets  better  --  to  get  as  much 
dedication,  as  much  effectiveness,  out  of  our  people  as 
possible.  In  spite  of  the  present  environment,  we  must 
continue  to  find  legitimate  ways  to  attract  the  manpower 
we  need  to  keep  our  forces  viable. 

It’s  been  estimated  that  maintaining  an  all-volunteer 
force  of  2.2  million  over  the  long  haul  will  require  one 
out  of  every  three  qualified  and  available  men  in  this 
country  to  volunteer  for  active  military  service.  Under 
present  policies,  one- third  will  have  to  be  "above  average," 
over  one-half,  "average."  This  year  alone  the  armed  services 
will  have  to  attract  better  than  400,000  young  Americans  to 
meet  our  commitments.  But  r.he  problem  goes  beyond  numbers. 
They  also  have  to  be  the  right  kind  of  people.  Here  is 
where  we've  got  some  room  for  improvement. 
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For  example,  out  of  the  average  100  enlistees  who  enter 
the  Air  Force,  25  leave  before  they  complete  4 years.  Some 
are  found  unsuitable  or  unfit  for  military  service;  some 
have  disciplinary  problems;  others  leave  for  personal  reasons. 
In  addition  to  the  personal  anguish  involved,  these  people 
represent  time,  effort,  and  money,  all  of  which  are  in  short 
supply  these  days.  We  know  that  high  school  graduates  have 
fewer  disciplinary  problems  than  non-graduates,  by  a ratio 
of  one  to  four.  (Yet  we  know  that  the  majority  of  non- 
graduates can  do  well.) 

The  question  is  why?  What  are  the  factors  working  in 
this  area?  Can  we,  can  you,  identify  them?  Is  it  something 
we're  doing?  Should  we  try  to  deal  with  these  people  before 
they  come  on  board  or  after,  or  both?  These  are  tough 
questions;  but  they  need  to  be  addressed,  and  I look  to 
you  for  the  answers . 

After  we  get  the  right  kind  of  people,  we  also  have  to 
manage  them  properly.  I sincerely  believe  young  people  today 
are  as  eager  as  ever  to  do  a good  job.  I also  know  they  want 
to  ask  questions.  They  v;ant  to  know  why  as  well  as  what;  and, 
above  all,  they  want  to  be  challenged  and  provided  the 
opportunity  to  participate  in  decisions  that  affect  their 
future. 
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This  means  we  have  to  avoid  arbitrarily  poking  these 


people  into  holes.  We  have  to  realize  and  act  on  the 
premise  that  what  may  excite  and  challenge  one  man  or 
woman  may  not  challenge  another.  In  the  long  run, 
arbitrary,  "convenient"  actions  can  create  a situation 
in  which  an  individual  who  might  have  been  a tremendous 
success  ends  up  being  bored  and  indifferent  to  his  work 
or  elects  to  leave  us. 

Our  need  here  is  obvious.  We  must  have  the  best  tools 
we  can  get  to  classify  and  assign  people.  We  must  have  the 
best  predictors  of  success  in  training  and  on  the  job  that 
you  can  give  us.  We  must  be  as  sure  as  we  are  able  that 
the  system  is  doing  everything  it  can  to  provide  each  person 
who  comes  to  us  the  best  opportunity  possible  to  grow,  and 
learn,  and  progress  to  the  limits  of  his  or  her  capabilities. 

I would  like  to  underline  the  fact  that  this  philosophy 
applies  to  the  minority  as  well  as  the  majority  members  among 
us . 

Like  every  other  service,  the  Air  Force  is  trying  hard 
to  satisfy  the  expectations  of  its  minority  members.  We  have 
almost  250  enlisted  career  fields  in  the  Air  Force.  A little 
better  than  a year  ago,  91  of  them  had  less  than  a 5 percent 
minority  population.  Today  that  number  is  down  to  about  45, 
and  we  are  heading  for  0.  Like  the  other  services,  we  are 
working  the  problem  and  the  effort  is  starting  to  pay  dividends. 
We  still  can  use  help,  all  the  help  you  can  give  us. 
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What  I have  been  discussing  so  far  is  just  a small 
fraction  of  the  challenge  we  face.  Even  this  incomplete 
list,  however,  indicates  the  need  today  for  aggressive, 
decisive  action.  This  is  the  time  for  the  military  to 
question,  to  search,  to  find  new  and  better  ways  to  manage 
its  human  resources,  to  take  the  lead  in  making  its  own 
unique  contributions  to  e growth  and  development  of 
military  people  as  vital  members  of  society. 

We  must  ask  ourselves.  What  are  the  implications  for 
testing?  What  role  should  tests  play  in  this  coming  decade? 

Of  all  the  many  specific  uses  to  which  tests  are  put,  is  there 
any  one  direction  or  unifying  theme?  I believe  there  is.  I 
think  there  is  a clear  direction  in  which  testing  should 
develop,  and  I think  there  is  an  underlying  theme  for  that 
direction. 

It  seems  to  me  that  the  requirement  to  get  maximum  value 
out  of  our  people  — all  our  people  — and  the  new  emphasis  on 
individual  challenge  and  choice  dictate  that  we  move  testing 
out  of  our  impersonal  "test  'em  and  tell  'em"  mode  of  the 
last  generation.  The  change,  in  my  view,  is  long  overdue. 
Whatever  reasons  required  managing  people  almost  solely  by 
testing  in  wartime  or  periods  of  rapid  mobilization,  the 
competitive  demands  of  a free-choice,  peacetime  labor  market 
are  different  and  they  require  different  solutions. 
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Which  means  what,  specifically? 

First,  I think  it  means  that  tests  of  the  future  must 
focus  much  more  on  the  individual,  on  his  unique  talents 
and  desires. 

Second,  it  means  fitting  tests  into  a scheme  that 
maximizes  the  opportunity  for  choice,  both  by  the  individual 
and  by  his  employer. 

Third,  as  a function  of  the  first  two,  it  means  that  tests 
must  provide  broader  profiles  of  information.  Four  or  five 
aptitude  scores  do  not  provide  a sufficient  basis  for  the 
kind  of  complex,  comprehensive  career  programs  we  have  today 
and  will  need  in  the  future. 

The  role  of  tests,  now  and  in  the  future,  should  be 
to  supply  a vital  piece  of  the  information  we  need  in  properly 
managing  the  careers  of  our  people.  The  quality  and  breadth 
of  this  information  will  be  crucial  to  the  quality  of  the 
decisions  we  make,  but  tests  must  not  serve  to  prejudge  the 
individual  for  a full  career.  They  should  clearly  be  a tool 
in  the  decision  process.  By  that  I mean  that  they  should  show 
the  implications  of  various  alternative  decisions. 

For  example,  tests  are  needed,  especially  with  the  advent 
of  the  new  instruction  systems,  that  can  recommend  specific 
kinds  of  learning  experiences  that  can  be  applied  to  an 
individual  case.  Lee  Brokaw,  your  program  chairman,  put  it 
well  recently  when  he  said,  "We  must  shift  emphasis  from  finding 
the  best  individual  for  the  job  to  find  the  best  job  for  the 
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New  systems  are  also  needed  to  provide  a more 
balanced  role  for  tests,  roles  that  clarify  where  tests 
fit  in  career  development,  roles  that  make  sure  the 
limitations  of  testing  are  equally  clear. 

We  also  have  to  remember  that  testing  policies  and 
practices  must  have  a relationship  to  this  nation's  goals 
in  the  area  of  social  policy.  The  Civil  Rights  Act  of  1966, 
the  Supreme  Court  decision  in  the  Duke  Power  Company  case, 
and  the  Equal  Employment  Opportunity  Act  of  1972  all  really 
make  one  point.  Equal  opportunity  means  giving  all  Americans 
a chance,  and  this  chance  must  be  more  than  just  a chance  to 
fail . 

As  Lyndon  Johnson  said  in  the  last  address  he  made 
before  his  death,  "History  has  not  been  equal  for  all 
Americans."  We  recognize  now  that  tests  can  be  used 
negatively.  They  can  perpetuate  a cycle  of  assignment  to 
jobs  with  limited  growth  potential.  They  can  confirm  the 
"inequality  of  history,"  in  Lyndon  Johnson's  terms,  by 
making  past  dif f iciencies  the  basis  for  continued  unfairness. 
In  short,  they  can  become  sentences  instead  of  doors  to  wider 
growth  and  development. 
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Ladies  and  gentlemen,  if  we  are  to  have  viable  armed 
forces  in  the  future,  we  must  come  up  with  reasonable  answers 
to  the  questions  and  problems  confronting  us  in  personnel 
management.  You  know  our  requirements;  you  know  how  urgent 
they  are . 

Today,  as  your  keynote  speaker,  I ask  you  to  provide  us 
the  tools  we  need  to  satisfy  these  requirements.  When  you 
bring  us  the  tools,  tell  us  how  to  use  them  and  how  not  to 
use  them,  and  make  sure  we  listen . As  a manager  with  a 
primary  interest  in  people,  I can  tell  you  we  ^ need  help, 
the  kind  of  help  only  you  can  provide.  I ask  that  you  come 
forward  and  tell  us  what  should  be  done. 

For  the  short  term,  give  us  the  best  answers  you  can. 
Remember,  we  can't  always  wait  for  the  research  effort  to 
be  neatly  organized,  the  right  researcher  to  be  found,  the 
research  effort  to  be  conducted  in  a simon  pure  environment, 
etc.  Very  often  we  need  answers  now . I challenge  you  to 
respond  quickly  when  the  situation  demands. 

For  the  long  term,  identify  the  areas  where  we  should 
encourage  further  research:  research  that  is  relevant; 

research  designed  to  tackle  the  tough  problems  of  tomorrow, 
not  yesterday's  problems;  research  that  will  enable  us  to 
achieve  progress  that  is  real  and  enduring. 
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And  finally,  let  the  experts  and  the  managers  (you 
and  I)  join  forces,  with  the  knowledge  that  there  is 
immediacy  and  urgency  in  the  work  we  do,  that  there  is 
excitement  and  opportunity  in  improving  the  management 
of  the  most  precious  asset  entrusted  to  any  organization. 
Now  may  not  be  the  best  of  times,  but  it  our  time. 
Let's  use  it  wisely. 
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Synopsi s 

KTA  Symposium  - Cooperative  Aptitude  Research 
Programs  Conducted  through  the  Armed 
Forces  Vocational  Testing  Group 


Parti cipants : 

Dr.  Harry  D.  Wilfong,  AFVTG 

Maj  Verna  S.  Kellogg  II,  AFVTG 

Capt  Melvin  T.  Gambrel  1,  Jr.,  AFVTG 

Dr.  Lonnie  D.  Valentine,  AFHRL  - Chairman 

Introductory  Comments  (Dr.  Valentine) 

1.  The  Armed  Services  High  School  Testing  Program  has  gradually 
evolved  since  1966  into  a program  of  sizable  importance. 

2.  In  today's  all  volunteer  era,  it  will  probably  assume  greater 
importance  to  the  services  as  a constructive  point  of  contact  between 
the  services  and  the  educational  community  with  its  pool  of  young  people 
entering  military  age. 

3.  For  about  the  past  6 months,  the  High  School  Testing  Program 
has  been  under  management  of  the  Armed  Forces  Vocational  Testing  Group 
at  Randolph  Air  Force  Base.  Other  participants  in  today's  symposium 
are  from  that  organization. 

4.  After  the  participants  have  each  briefly  discussed  some 
aspect  of  the  program,  the  floor  will  be  open  for  audience  questions 
and  suggestions.  It  is  hoped  that  the  discussion  period  will  serve 
to  elicit  constructive  suggestions  for  the  program. 

Program  History  (Major  Kellogg) 

1.  A popular  stereotype  of  the  recruiter  is  the  "lifer"  body- 
snatching  young  men  from  the  ghetto  and  the  juvenile  courts  and 
channeling  them  to  the  rehabilitation  environment  of  the  military 
services.  Nothing  could  be  further  from  the  truth.  The  services 
need  "quality"  human  resources --mi  1 i tary  services  today  demand 
trainable,  educable  and  alert  individuals. 

2.  The  sophistication  of  today's  weapon  systems  requires  the 
military  services  to  expend  more  time  and  money  training  a man  to 
an  acceptable  skill  level  than  they  can  anticipate  realizing  from 
his  services.  Thousands  of  young  people  leave  the  military  service 
after  their  first  enlistment.  Return  on  the  taxpayers'  investment  in 
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them  is  realized  mainly  in  terms  of  the  skills  they  take  back  into 
civilian  jobs. 

3.  Because  of  this  requirement  for  quality  manning,  the 
military  services  have  tied  their  recruiting  and  enlistment  programs 
directly  to  the  measurement  of  aptitude  for  vocational  training. 

4.  The  Department  of  Defense,  this  year,  directed  the  formation 
of  the  Armed  Forces  Vocational  Testing  Group--to  direct,  manage 

and  guide  the  Department  of  Defense  High  School  Testing  Program. 

This  program  is  the  cornerstone  of  recruiting  the  all-volunteer 
force. 

5.  The  primary  tool  of  the  testing  program  is  the  Armed 
Services  Vocational  Aptitude  Battery,  coimonly  referred  to  as 
ASVAB. 

6.  ASVAB  is  the  product  of  over  30  years  research  and 
development  in  the  area  of  military  classification--most  of  it 
accomplished  by  people  in  this  room.  Today's  enlistee  is  carefully 
tested  for  identification  of  his  potential  for  training. 

7.  The  genesis  of  the  High  School  Testing  Program  goes  back 
to  1958,  when  the  Air  Force  recruiters  struck  on  the  idea  of  going 
into  high  schools  with  a Vocational  Aptitude  Battery,  enabling 
them  to  identify  specific  youngsters  with  specific  aptitudes  as 
potential  recruits.  With  the  success  of  the  Air  Force  Program,  the 
other  services  initiated  similar  programs.  The  competition  caused 
confusion  and  projected  an  unprofessional  image  of  the  military 
services. 

8.  In  1966,  the  Department  of  Defense  directed  that  the  services 
combine  their  high  school  efforts. 

9.  A group  of  service  behavioral  scientists  met  and  the 
eventual  fruit  of  their  efforts  was  ASVAB--ASVAB  combined  the  best 
and  most  desired  features  of  all  the  military  classification  tests 
and  gave  the  High  School  Testing  Program  a product  usable  by  all 
the  services. 

10.  The  concept  was  good--but  different  parts  of  the  action 
were  assigned  to  the  various  servi ces--the  Army  was  tasked  to 
develop  the  test,  the  Navy  was  tasked  to  provide  counselor's 
materials  and  the  Air  Force  supervised  the  scoring  and  return  of 
the  test  to  the  counselor.  Recruiters  went  back  into  the  schools, 
all  using  the  same  test,  but  still  with  their  parochial  interests 
at  heart.  Thus,  ASVAB  became  an  Army  test,  a Navy  test  or  an 
Air  Force  test  depending  upon  the  representative  who  got  there  first. 
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Both  recruiters  and  counselors  lost  sight  of  the  value  of  the 
battery  in  helping  young  people  plan  their  futures — and  got 
hung  up  on  the  idea  that  ASVAB  was  only  a means  to  get  names 
of  potential  enlistees. 

11.  The  Armed  Forces  Vocational  Testing  Group  is  the  answer 
to  the  problem.  Staffed  by  personnel  of  all  the  military 
services — Army,  Navy,  Marines  and  Air  Force--the  group  provides 

a single  manager  and  interpreter  of  ASVAB  for  the  Department 
of  Defense  Fligh  School  Testing  Program.  Dealing  with  the  local 
recruiter  through  an  interservice  recruitment  committee  (organized 
by  geographical  location  and  representing  each  service)  the  group 
assures  equitable  and  professional  administration  of  ASVAB. 

12.  The  mission  of  the  Armed  Forces  Vocational  Testing  Group 
is  to  place  ASVAB  in  the  schools,  assure  its  currency  and  validity 
and  provide  the  student  and  counselor  with  assistance  and 
infoniiation.  Our  job  ends  there;  if  the  youngster  is  interested 
in  the  military  as  a means  to  vocational  training,  the  recruiter 
provides  him  with  that  information  after  the  test  has  been 
administered. 

13.  Overt  recruiting  is  not  allowed  in  conjunction  with  either 
the  administration  of  the  battery  or  the  delivery  of  the  scores  to 
the  counselor  and  the  student.  Improved  recruiting  is  an  antici- 
pated side  effect  in  those  instances  where  aptitude  and  service 
interest  are  demonstrated. 

Armed  Services  Vocational  Aptitude  Test  (Capt  Gambrell) 

1.  Over  the  past  30  years,  the  various  military  services 
have  used  aptitude  tests  to  select,  classify  and  place  enlistees. 

As  Major  Kellogg  has  previously  mentioned,  in  1958,  the  Air  Force 
conceived  the  idea  of  going  directly  to  the  high  schools  with  an 
aptitude  test,  the  AQE.  This  proved  to  be  a most  successful  program 
for  pre-qual ifying  prospects  for  enlistment.  The  other  services 
soon  adopted  similar  programs  and  high  school  aptitude  testing 

by  military  services  became  a reality. 

2.  I would  like  to  direct  niy  section  of  this  panel  to  a 
review  of  development  of  the  ASVAB,  content  and  standardization. 

The  primary  purpose  of  this  review  is  to  reacquaint  you  with  the 
basic  structure  of  our  current  ASVAB. 

3.  The  initial  battery  development  study  (Bayroff  & Fuchs, 
1970)  was  directed  toward  (A)  identifying  interchangeable  tests 
within  the  existing  service  classification  batteries,  (B)  developing 
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tests  for  the  joint  battery  from  a pool  of  test  items  composed  of 
those  in  the  interchangeable  components  of  the  existing  service 
batteries  and  (C)  standardization  of  the  resultant  battery. 

4.  To  establish  test  interchangeability,  all  of  the  service 
classification  batteries  were  administered  to  a common  sample  of 
3900  basic  trainees  from  the  Arniy,  Navy,  Air  Force  and  Marine 
Corps.  Testing  was  arranged  for  three  separate  days, 

with  only  one  of  the  service  batteries  being  administered  on  a given 
day,  and  battery  administration  was  counterbalanced  to  eliminate 
effects  of  test  practice. 

5.  A stratified  subsample,  representative  of  the  full  range 
of  abilit,’  on  the  Armed  Forces  Qualification  Test,  was  used  to 
establish  intercorrelations  among  all  of  the  tests  in  the  various 
service  classification  batteries;  this  resulted  in  the  identification 
of  seven  interchangeable  content  areas.  Two  other  areas  were 

added  to  ASVAB  to  yield  an  AFQT  equivalent. 

6.  The  nine  ASVAB  tests  yield  five  aptitude  composites  which 
are  reported  to  high  school  counselors  through  standard  format. 

These  are: 

(A)  Electronics  - describes  students  in  terms  of  abilities 
relevant  to  electrical/electronic  occupations.  The  aptitude  score 
consists  of  tests  dealing  with  electrical  information  and  understanding 
mechanical  principles. 

(B)  General  Maintenance  - describes  capabilities  relevant 
to  a variety  of  mechanical  and  trade  jobs.  The  composite  consists 
of  tests  assessing  shop  information  and  spacial  ability. 

(C)  Motor  Maintenance  - is  concerned  with  ability  for 
engine  repair  and  related  jobs  and  measures  automotive  information  and 
understanding  of  mechanical  principles. 

(D)  Clerical  - measures  ability  relative  to  clerical/ 
administrative  occupations.  Composite  concerned  with  verbal  ability 
and  clerical  speed  and  accuracy. 

(E)  General  Technical  - describes  students'  ability  for 
occupations  requiring  academic  ability.  This  aptitude  is  taken  from 
verbal  and  mathematical  components  of  battery. 

7.  The  battery  contains  300  test  items;  100  in  the  Coding 
Speed  and  25  in  each  of  the  remaining  eight  tests.  Actual  testing 
time  is  112  minutes;  generally  two  and  one-half  hours  are  required 
to  administer  the  battery.  Speed  is  not  emphasized,  although  all 
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losts  are  timed. 

8.  Te>;t  items  contained  in  Form  1 of  the  ASVAB  (v/ith  the  exception  of 
those  in  Coding  Speed)  were  selected  from  an  item  pool  consisting  of 

a'l  items  cortained  in  the  service  classification  batteries  used  in 
the  'interchangeabi  li  ty"  study.  Criteria  for  item  selection  v^ere 
difficulty  level  (proportion  of  examinees  responding  correctly), 
discrimination  level  (ability  of  the  item  to  discriminate  correctly 
between  individuals  who  score  high  and  those  who  score  low  on  the 
relevart  ability  as  n^flected  in  the  item's  correlation  with  other 
items  of  its  type),  and  content  validity. 

9.  The  25  items  in  each  subtest  (except  coding  speed)  were 
ari^ariged  nt  approximate  difficulty  order  so  r.hat  about  85  percent 
of  examinees  would  answer  the  first  few  items  correctly,  with  the 
success  oercentage  decreasing  to  about  25  percent  at  the  end  of  each 
subtest. 

10.  T^e  first  form  of  the  ASVAB  was  stanoardized  on  stratified 
sample  tested  at  a geographically  representative  sample  of  11  Armed 
Forces  Fxarining  and  Entrance  Stations  (AFEESs).  The  stratified 
sample  was  designed  to  be  representative  of  the  scores  obteined 

by  the  irobi  li zation  population  on  the  AFQT. 

11.  Forms  2 and  3 of  the  ASVAB  were  developed  from  a pool  of  new 
items,  but  procedures  followed  in  their  development  and  standardization 
are  very  similar  to  those  for  Form  1. 

12.  The  reliabilities  and  validities  of  the  battery  have  been  found 
to  be  quite  satisfactory,  Kuder-'lichardson  reliabilities  range  from  .78 
to  .87.  The  validity  coefficients  for  service  technical  school  courses 
art  about  equal  to  those  of  the  earlier  classification  batteries. 

ASVAB  Research  (Dr,  Wilfong) 

1.  Since  the  history,  development  and  content  of  the  ASVAB 
liave  already  been  thoroughly  covered  by  Major  Kellogg  and  Captain 
Gambrel  1 , I shall  limit  niy  comnents  to  a brief  description  of  research, 
both  on-going  and  projected,  conducted  in  support  of  the  program, 

2.  Our  immediate  efforts  ere  directed  toward  increasing  the 
usability  and  application  of  the  ASVAB  counselor's  manual.  Toward 
this  end — the  normative  base  is  being  expanded  and  a broader  sampling 
of  military/civilian  occupations  will  be  included  in  Volume  2. 

3.  From  the  historical  tape  on  ASVAB  maintained  at  AFHRL, 
lackland  AFB,  the  Personnel  Research  Laboratory  has  extracted  a 
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representati ve  sampling  of  36  thousand  cases  producing  separate  norms 
by  geographical  region,  sex  and  high  school  grade.  It  has  been  niy 
experience,  thus  far,  that  these  are  the  three  most  common  comparative 
variables  requested  by  high  school  counselors. 

4.  The  other  ASVAB  research  requirement  is  to  expand  the 
listing  of  civilian/military  occupations  as  cataloged  in  Volume  2 

of  the  counselor's  manual.  What  is  being  planned  - assuming  sufficient 
lead-time  to  permit  publication  for  the  1974-75  school  year--as  a 
two-columinar  hierarchical  listing  of  representative  civilian  and 
service  jobs  anchored  to  appropriate  dictionary  of  occupational  titles 
numbers . 

5.  Long-range  research  needs  identified  in  support  of  the  ASVAB 
program  are  too  numerous  to  cite  during  this  brief  presentation. 

However,  I would  like  to  touch  upon  some  of  the  more  significant  efforts 
that  are  planned. 

6.  First,  content  revisions  are  being  considered,  including 
deletion  of  the  tool  knowledge  subtest,  possible  shortening  of 
some  of  the  other  portions  of  ASVAB  and  the  insertion  of  an 
occupational  interest  schedule.  These  changes  should  appear  in 
Forms  5 and  6 of  the  ASVAB,  hopefully,  to  be  available  by  the 
1975-76  school  year. 

7.  It  is  the  general  consensus  of  researchers  exposed  to  the 
ASVAB  that  a highly  desirable  research  undertaking  would  be  development 
of  current  mobilization  base  data--representative  of  modern  America 

on  as  many  strata  as  possible.  What  is  envisioned  is  a study  broad 
enough  to  serve  many  masters  and  the  needs  of  many  agencies  while 
simultaneously  substituting  for  the  old  1942-44  mobilization  data 
base.  The  many  problems  involved  in  planning,  securing  approval 
for,  and  executing  such  an  ambitious  undertaking  are  well  recognized. 

8.  Another,  equally  critical,  avenue  for  research  exploration 
is  to  establish  a mechanism  for  conduct  of  a series  of  longitudinal 
validation  studies  tracking  students  at  successive  stages  along  their 
career  path,  both  within  and  outside  of  the  military  service.  By 

its  very  nature,  the  ASVAB  program  presents  the  ideal  vehicle  for 
performing  such  research. 

9.  A wide  scenario  of  research  support  requirements  has  been 
identified  for  the  ASVAB  program,  including  subjects  as  listed  in 
Attachment  1,  Armed  Services  Vocational  Aptitude  Testing  Research 
and  Development  Support  requirements,  of  the  joint  services 
regulation  covering  the  Armed  Forces  Vocational  Testing  Program. 
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10.  Research  needs  identified  by  high  school  counseling  personnel 

and  by  service  recruiting,  production  and  classification  representatives 
range  across  a broad  spectrum,  covering  such  topics  as:  tracing  the 

longitudinal  career  decision  process;  exploring  the  feasibility  of 
remote  test  scoring  and  concurrent  feedback;  establishing  a task/ 
aptitude  criterion  matrix,  permitting,  in  turn,  rapid  estimation  of 
predictive  validities  for  new  jobs  entering  the  inventory;  reconfigured 
jobs,  or  jobs  having  an  insufficient  N for  traditional  validation 
purposes;  and  a linear  programmed  approach  to  test  administration. 

11.  Rather  than  describe  the  entire  spectrum  of  our  anticipated 
research  objectives  at  this  time,  I would  prefer  to  return  the  floor 
to  the  workshop  chariman  so  that  we  can  explore  subjects  in  which 
you  may  have  a particular  interest. 

Concluding  Comments  (Dr.  Valentine) 

1.  My  few  conments  represent  my  personal  views--not  necessarily 
policy  of  AFHRL  or  the  Air  Force. 

2.  AFHRL's  role,  as  well  as  that  of  the  other  service  R&D  labs, 
in  the  program  is  viewed  as  one  requiring  sound  R&D  to  support  and 
improve  the  program.  Policy  with  regard  to  the  program  is  the 
responsibility  of  service  and  DOD  Hq  agencies. 

3.  There  are  three  areas  in  which  substantative  support  R&D 
is  needed.  These  deal  with  (a)  breadth  of  battery  content  and 
coverage,  (b)  battery  standards  (which  include  both  norming  and 
composite  structure),  and  (c)  validity  studies. 

4.  In  its  present  form,  ASVAB  is  limited  to  subtests  either 
"common"  to  earlier  service  classification  batteries  or  necessary  to 
provide  an  AFQT  score.  Most  of  these  predecessor  batteries  were 
designed  to  provide  optimal  prediction  of  training  success.  Such 
composites  tend  to  intercorrelate  substantially.  School  counselors 
have  always  been  constrained  to  counsel  all  students,  and  the  optimal 
test  system  for  them  both  identifies  ability  level  and  provides  optimally 
different  classification  indexes.  The  services  have  been  able 

to  select  rather  than  to  simply  classify  in  the  true  sense. 

Under  the  volunteer  force  the  service-  are  going  to  be  compelled 
to  join  counselors  in  a more  effective  job  of  differential 
classification.  In  addition,  the  ASVAR's  present  limited  content 
coverage  short-changes  some  of  the  classification  needs  of  some 
of  the  services.  Ihus,  the  service  labs  need  to  give  careful 
attention  to  appropriate  added  content  and  to  the  classification 
models  which  will  allow  decisions  about  both  level  of  aptitude 
and  the  differential  aptitude  demands  of  jobs. 
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5.  The  mobilization  population  standards  against  which  the 
services  typically  calibrate  new  tests  is  now  30  years  old,  and  it 
does  not  take  into  account  special  subgroups  of  particular  interest 
for  special  programs.  A broad  based  national  normative  census, 
with  provision  for  identification  of  standards  appropriate  to 
particular  population  subgroups,  is  needed.  Some  counselors  have 
expressed  dissatisfaction  with  the  service's  1942  mobilization  base 
standard  and  have  maintained  that  they  want  standards  based  on  high 
school  age  youth. 

6.  Substantative  ASVAB  validation  studies  not  only  against 
service  criteria  but  against  civilian  career  criteria  as  well,  are 
needed. 

7.  Careful  attention  to  these  three  areas,  with  cooperative 
planning  and  research  by  the  various  service  labs,  can  lead  to  development 
of  a comprehensive  and  flexible  test  battery  capable  of  satisfying 

both  service  and  counselor  needs,  and  can  be  an  invaluable  recruiting  aid 
to  all  the  services.  It's  accomplishment  will,  however,  require  that 
the  services  lay  aside  parochial  interests,  support  the  appropriate 
R&D  labs  with  the  manpower  and  other  resources  needed  for  its 
accomplishment,  and  encourage  long  range  cooperative  planning  and 
research. 

(The  symposium  was  concluded  with  an  audience  question  and 
reaction  period.) 


DEVELOPMENT  OF  THE  ARMED  SERVICES 
VOCATIONAL  APTITUDE  BATTERY 


M.  A.  Fischl,  Ph.D. 

U,  S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

The  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  is  the  test 
battery  administered  in  the  joint  services  high  school  testing  program. 

In  this  program  the  services  offer  to  administer  a battery  of  vocational 
aptitude  tests  in  any  high  school  that  requests  it.  Trained  test  adminis- 
trators visit  the  school  and  administer  the  battery,-  the  tests  are  scored 
centrally,  and  results  are  returned  to  the  high  school  guidance  counselor 
for  transmittal  to  the  student.  This  program  is  provided  to  high  schools 
that  request  it,  at  no  charge  either  to  the  school  or  to  the  student. 
Identification  of  Content 

The  tests  employed  in  this  program  were  developed  to  be  a set  of 
short  alternate  forms  of  the  common  elements  of  classification  test  batteries 
in  use  by  the  services.  At  the  time  of  the  original  development  each  of  the 
services  was  administering  a classification  battery  of  nine  or  ten  aptitude 
tests.  To  Identify  the  common  content  material  of  these  batteries,  the  batter- 
ies of  the  Army,  Navy,  and  Air  Force  were  administered  to  a national  sample  of 
3900  enlisted  men  during  their  first  few  days  of  service.  Each  enlisted  man 
took  the  tests  of  all  three  batteries,  in  a counterbalanced  sequence,  one 
battery  a day,  over  a several  day  period.  The  sample  was  subjected  to  statisti- 
cal adjustment  to  equate  for  differences  in  screening  methods  and  acceptance 
standards  among  the  services,  and  the  test  scores  were  Intercorrelated 
to  determine  which  test  content  areas  were  sufficiently  similar  as 
to  comprise  the  core  of  a joint  test  battery.  The  criterion  of  similarity 
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was  set  as  a correlation  coefficient  of  0.90i^’  and  this  analysis  identi- 
fied the  seven  test  content  areas  of  verbal  fluency,  quantitative 
reasoning  ability,  spatial  visualization,  mechanical  comprehension,  shop 
information,  automotive  Information,  and  electronics  information.  An 
eighth  content  area,  clerical  ability,  was  added  because  of  its  impor- 
tance for  counseling  and  classification,  even  though  the  service  tests  of 
this  function  did  not  meet  the  statistical  criterion.  Finally,  knowledge 
about  hand  tools  was  added  as  a content  area  so  that  all  four  areas 
assessed  in  the  Armed  Forces  Qualification  Test  (AFQT)  would  be  represented 
in  ASVAB. 

Construction  of  the  Original  Tests 

For  all  but  the  clerical  content  area,  25  items,  spanning  a full 
range  cf  difficulty,  were  selected  from  the  operational  tests  of  the 
services.  These  items  were  assembled  so  as  to  comprise  new  forms  of  the 
service  tests  in  the  respective  content  areas.  The  clerical  ability  test 
was  assembled  by  taking,  Intact,  the  most  valid  of  the  service  tests  in  ■ 
this  area.  It  is  a 100-itera  speed  test;  all  of  the  others  are  power  tests 
and  are,  with  the  exception  of  iool  Knowledge,  half-length  forms  of  the 
service  tests  in  the  counterpart  areas.  The  Tool  Knowledge  test  contains 
the  same  number  of  items  as  was  contained  in  the  Tool  Knowledge  portion 
cf  the  then  operational  AFQT. 

Exhibit  1 describes  the  contents  of  the  nine  tests  of  the  battery. 

To  obtain  norms,  these  tests  were  administered  to  a national  sample 

Insert  Exhibit  I about  here 

2./  Corrected  for  restriction  of  range  and  lack  of  perfect  reliability. 
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EXHIBIT  I 


TESTS  IN  THE 

ARMED  SERVICES  VOCATIONAL  APTITUDE  BATTERY 


1.  CODING  SPEED  TEST  (CS).  IN  THTS  TEST  THERE  IS  A KEY  AND  100  ITEMS. 

THE  KEY  IS  A GROUP  OF  WORDS  WITH  A CODE  NUMBER  FOR  EACH  WORD.  EACH 
ITEM  PRESENTS  ONE  WORD  FOR  WHICH  THE  EXAMINEE  INDICATES  THE  CODE  • 
NUMBER. 

2.  WORD  KNOWLEDGE  CWK).  EACH  ITEM  REQUIRES  THE  EXAMINEE  TO  SELECT  THE 
CORRECT  SYNONYM  FOR  A SPECIFIED  WORD._ 

3.  ARITHMETIC  REASONIhJG  (AR).  EACH  ITEM  IS  A REASONING  PROBLEM  INVOLVING 
APPLICATION  OF  ARITHMETIC  PROCESSES. 

4.  TOOL  KNOWLEDGE  (TK) . EACH  ITEM  PRESENTS  FIVE  DRAWINGS  OF  VARIOUS 
TOOLS  OR  SHOP  EQUIPMENT.  THE  EXAMINEE  INDICATES  WHICH  OF  THE  FOUR 
ALTERNATIVE  DRAWINGS  GOES  BEST  WITH  THE  LEAD  DRAWING. 

5.  SPACE  PERCEPTION  (SP).  EACH  ITEM  CONSISTS  OF  5 DRAWINGS:  A PATTERN 

AND  FOUR  BOXES.  THE  QUESTION  TO  BE  ANSWERED  IS  WHICH  ONE  OF  BOXES 
CAN  BE  MADE  BY  FOLDING  THE  GIVEN  PATTERN. 

6.  AUTOMOTIVE  iNhORMATION  (Al).  EACH  HEM  ASKS  A QUESTION  ABOUT  THE 
IDENTIFICATION  OR  OPERATION  OF  AUTOMOBILE  PARTS.- 

7.  SHOP  INFORMATION  CSI).  THIS  TEST  HAS  QUESTIONS  ABOUT  SHOP  PRACTICES 
AND  THE  USE  OF  TOOLS.  MANY  OF  THE  ITEMS  CONTAIN  DRAWINGS. 

8.  MECHANICAL  COMPREHENSION  CMC).  EACH  ITEM  INCLUDES  A DRAWING  OR 
DRAWINGS  ILLUSTRATING  SOME  PHYSICAL  PRINCIPLE  AND  A QUESTION  CON- 
CERNING IT. 

9.  ELECTRONIC  INFORMATION  (ELI).  THIS  TEST  HAS  QUESTIONS  ABOUT  ELEMENTARY 
PRINCIPLES  OF  ELECTRICITY  AND  ABOUT  ELECTRICAL  OR  ELECTRONIC  DEVICES. 
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of  3050  Selective  Service  registrants,  stratified  to  represent  the 
population  of  young  men  of  military  age. 

Construction  of  Successor  Forms 

Coincident  with  introduction  of  the  original  form  of  the  battery  in 
the  school  year  of  1968-69,  work  began  to  develop  two  entirely  new  forms. 

For  each  of  the  eight  power  testsjL^  six  times  the  required  number  of  items 
were  prepared.  This  is  customary  procedure,  preparing  a substantial  overage 
in  anticipation  of  attrition  in  the  item  analysis  stage.  These  items  were 
administered  to  several  national  samples  stratified  to  represent  the  popu- 
lation of  military  age  young  men.  The  total  n^  was  4000  cases,  18%  of 
whom  were  blacks,  and  item  analysis  statistics  of  difficulty  and  homo- 
geneity were  obtained.  Using  these  statistics,  items  were  assembled  into 
two  parallel  25-item  forms  of  each  of  the  eight  power  tests.  Each  test 
was  constructed  to  be  of  wide  ranging  difficulty,  with  items  spanning 
virtually  the  entire  range  of  99  per  cent  difficulty  (very  easy)  to  1 
per  cent  difficulty  (very  hard)  in  the  reference  population.  When  coupled 
with  the  new  forms  of  the  speed  test,  there  were  two  entirely  new  nine- 
test  batteries. 

These  batteries  were  administered  to  several  stratified  national 
samples  of  Selective  Service  registrants,  one  form  to  a sample.  The 
total  n was  3500  cases,  virtually  all  whom  were  between  18  and  19  years 
of  age,  80%  of  whom  had  completed  between  10  and  13  years  of  education, 
and  15.5%  of  whom  were  black.  From  this  administration  percentile  and 
Army  Standard  Score  norms  were  developed,  and  test  reliability  coefficients, 
intercorrelations,  and  other  characteristics  were  derived. 
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— ^The  Speed  test  was  prepared  in 
pf  a speed  test  is  not  meaningful 


its  final  forms,  because  item  analysis 
, and  item  attrition  is  not  incurred. 


Characteristics  of  the  Battery 

The  entire  battery  requires  approximately  2 1/2  hours  to  administer. 

The  longest  test,  Arithmetic  Reasoning,  takes  about  25  minutes;  the 
shortest.  Coding  Speed  (clerical  aptitude^  takes  7 minutes;  most  of  the 
others  are  10-minute  tests. 

Reliability  coefficients  are  presented  in  Table  1,  and  test  intercorre- 
lations appear  in  Table  2.  The  intercorrelations  are  of  intermediate 

Table  1 

ASVAB  Reliability:  Coefficients  of  Equivalence  and  Internal  Consistency 


for  Each  Test 


Test 

Equivalence  i/ 

Internal  Consistency  ZJ 

Coding  Speed 

v£> 

00 

00 

Not  Applicable 

Word  Knowledge 

.80,  .85 

.87 

Arithmetic  Reasoning 

.81,  .86 

.87 

Tool  Knowledge 

.79,  .76 

.78 

Space  Perception 

.82,  .84 

.84 

Mechanical  Comprehension 

.77,  .73 

.83 

Shop  Information 

.67,  .65 

.81 

Automotive  Information 

.75,  .74 

.85 

Electronics  Information 

.80,  .79 

.83 

1.  Correlation  of  Form  2 with  Form  3,  in  two  subsamples.  Form  2 
administered  first  in  one  (ii  = 536),  Form  3 administered  first  in  the 
other  (ji  = 514) . 

2.  Utilizing  Kuder-Richardson  formula  20,  ri  = 616. 

(insert  Table  2 here) 
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magniture,  with  the  perceptualmotor  test  (Coding  Speed)  being  generally 
most  Independent  of  the  rest  of  the  battery. 

The  reliability  coefficients  seem  reasonably  high  for  tests  of  only 
25-item  length,  and  predictive  accuracy  is  enhanced  operationally  by 
utilizing  combinations  of  several  tests.  Why  the  Shop  Information  test 
yielded  the  lowest  equivalence  coefficients  in  the  battery  is  not  apparent, 
especially  since  its  internal  consistency  coefficient  is  of  comparable 
magnitude  to  the  rest  of  the  set. 

Occupational  predictions  from  ASVAB  are  generally  made  utilizing 
combinations  of  several  tests.  These  combinations,  both  with  ASVAB  tests 
and  their  counterparts  in  the  services’  operational  batteries,  have  under- 
gone extensive  validation  study  against  criteria  of  training  success  in 
large  numbers  of  job  specialties.  The  modal  validity  coefficient  against 
these  criteria  has  been  of  the  order  0.60,  with  a range  of  0.35  to  0.75. 

In  summary,  the  ASVAB  is  a nine-test  aptitude  battery  developed  to  be 
shortened  alternate  forms  of  the  common  elements  of  the  then  service  classifica- 
tion batteries.  it  has  been  empirically  developed,  and  tried  out  on  large 
representative  samples  of  the  population  to  which  it  is  Intended  to  be 
applied.  The  original  form  was  in  use  in  the  joint  service  high  school 
program  for  five  years.  Form  2 has  been  operational  in  the  schools  since 
the  beginning  of  this  school  year,  and  testing  goals  are  in  excess  of  a 
million  students  by  the  end  of  the  year. 
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"NEW  DEVELOPMENTS  IN  DEFENSE  LANGUAGE  PROFICIENCY  TESTING" 

PRESENTED  BY  DR.  C.D.  LEATHEIiMAN  AND  DR.  A.  AL-HAIK  OF  THE 
DEFENSE  LANGUAGE  INSTITUTE 

1973  MILITARY  TESTING  ASSOCIATION  CONFERENCE,  SAN  ANTONIO,  TEXAS 

28  OCT  - 2 NOV  1973 


PURPOSE 

Di'finsc  LvTnguTp.e  Proficiency  Tests,  commonly  referred  to  as  DLPT's  arc  tests 
designed  to  m.easurc  the  language  proficiency  of  all  DLI  graduates,  as  well 
as  DOD-connectcd  individuals  claiming  proficiency  in  a particular  foreign 
language.  Specifically,  the  DEPT  serves  two  major  purposes: 

a.  To  measure  the  listening  comprehension  and  reading  comprcliens ion 
sl'.ills  in  a foreign  language.  These  two  skills  range  from  minimum  ability 
which  nay.be  useful  in  an  activity  such  as  directing  traffic,  to  the  skill 
needed  by  a competent  interpreter.  The  test  does  not  discriminate  among 
the  higher  language  skills  required  by  a top-level  simultaneous  translator. 

b.  To  evaluate  and  rate  the  examinee's  ability  to  meet  the  linguistic 
requirements  for  particular  jobs,  sucli  as  inter pre t or , interrogator,  and 
translator.  llie  test  is  not  intended  to  assess  other  skills,  such  as 
interrogation  techniques,  which  may  be  required  by  tficse  jobs. 


BACKGROUND 

Tlie  history  of  military  proficiency  testing  goes  hack  to  1948  when  the  Army 
first  developed  what  was  then  called  The  Army  Language  Proficiency  Tests. 
Tests  in  31  languages  were  developed  and  used  between  1948  and  1953.  Each 

test  consisted  of  four  parts:  Parts  I and  II  covered  understanding  of  the 

spoken  langimge;  Part  III  dealt  with  reading  ability;  and  Part  IV  measured 

writing  ability  through  knowledge  of  grammar.  Due  to  military  pressures  at 
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the  time,  the  development  of  these  tests  involved  only  a minimum  of 
research.  During  and  following  the  Korean  War  it  became  apparent  that 
the  tests  were  not  always  reliable  in  discriminating  between  individuals 
of  varying  levels  of  language  ability.  As  a result  of  tliis  recognized  defi- 
ciency, Army  user  agencies  requested  the  development  of  new  Army  Language 
Proficiency  Tests.  Accordingly,  in  195A  The  Adjutant  General's  Office  was 
directed  to  develtip  two  prototype  Army  Language  Proficiency  'Jests:  one  for 

an  Asiatic  ianguige,  namely,  Chincsc-Manda r in  and  one  for  a European  lan- 
guage, Russian.  Using  these  tests  as  guides  the  Army  Liinguage  School 
(the  predecessor  of  The  Defense  Language  Institute)  was  subsequently  directed 
ti)  prepare  language  tests,  botli  to  replace  existing  instruments  and  to  develop 
new  tests 'in  languages  ior  which  there  were  no  tests  at  that  time.  Even 
thougli  the  new  prototype  tests  in  Russian  and  Chinese-Mandarin  were  carefully 
developed  and  validated,  the  other  new  tests,  approximately  38  in  number, 
consisted  mainly  of  translations  of  the  two  prototype  tests  and  were  not 
subjected  to  systematic  validation  procedures  before  their  introduction  into 
tlic  Defense  Language  Program  in  1958.  As  a result,  Russian  proficiency  Lest 
was  used  as  the  model  for  all  European  languages  and  the  Chinese-Mandarin 
test  w.is  used  for  all  Asiatic  languages.  It  became  apparent  following 
extensive  use  during  the  subsequent  years  that  these  38  additional  tests  were 
not  as  valid  or  as  reliable  as  had  been  assumed.  Additionally,  the  use  of  a 
single  test  fi'im  for  eacli  of  these  tests  over  more  than  a decade  further 
reduced  their  effectiveness  due  to  possible  test  compromise.  Recognizing  these 
deficiencies.  The  Defense  Language  Institute  initiated  a series  of  new  projects 
in  1966  aimed  at  developing  follow-on  DLPT  IIs,  or  second  generation  DLPTs  to 
replace  the  old  ones.  The  first  group  of  DLPT  IIs,  were  developed  under 


A 


28 


contract  with  the  Educational  Testing  Service,  utilizing  language  experts 
from  the  DLI  and  from  various  universities,  in  addition  to  resources  from 
KTS.  Seven  high-enrollment  languages  were  included,  such  as  Russian, 

Spanish,  Arabic,  etc.  These  initial  DLPT  IIs  are  introduced  into  the  Defense 
Language  Program  at  the  present  time.  As  these  tests  are  introduced  into  the 
system  and  prove  to  be  operationally  sound,  other  language  proficiency  tests 
and  alternate  forms  will  then  be  introduced  into  the  system.  Current  plans 
call  for  the  development  of  two  or  three  alternate  forms  as  a minimum  for 
each  high  density  DLPT  II.  DLI's  R&D  plan  includes  the  construction  and 
validation  of  a computer  bank  of  test  items  for  each  language,  from  which 
multiple  test  forms  can  be  assembled  automatically  and  actually  produced 
in  final  form,  botli  test  booklets  and  audio  tapes,  by  computer.  These  were 
the  facts  and  some  of  the  events  which  led  to  our  current  efforts  with  regard 
to  second  generation  DLPTs.  Additional  details  concerning  specific  aspects 
of  the  developmental  work  program  are  given  below. 

DEVELOPMl-N'T  OF  PROTOTYPE  TESTS  IN  RUSSIAN  AND  CH INliSE-MANDARIN 

These  early  prototype  tests  were  developed  by  Army  Language  School  personnel, 

in  cooperation  with  representatives  from  the  Personnel  Research  Branch  of  Ttie 

Adjutant  General's  Office.  Each  test  has  two  parts:  Listening  Comprehension, 

consisting  of  sixty  items  and  Reading  Comprehension  also  consisting  of  sixty 

items.  All  items  are  multiple-choice  type,  each  consisting  of  a lead  sentence 

or  a short  paragraph  in  the  foreign  language  and  four  optional  responses 

printed  in  English  in  the  test  booklet.  Tliree  of  these  options  are  plausible 

distractors  whereas,  only  one  is  the  correct  answer.  The  options  can  be  words, 

phrases,  sentence  translations,  or  paragraph  gistings  depending  on  the  objec- 

Listening  comprehension  item  leads  are  voiced  on  a tape, 
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live  of  the  item. 


but  reading  comprehension  item  leads  are  printed  in  the  test  booklet.  After 
completion,  the  tests  were  field  Le.sted,  including  a validation  measure,  with 
Army  personnel  possessing  varying  degrees  of  skill  and  experience  in  the  two 
languages.  The  validation  instrument  consisted  of  job-oriented  tasks  such 
as  interpreting  and  translating.  The  median  correlation  between  tlie  Chinese 
test  and  the  criterion  measure  w.is  .70  and  between  tlie  Russian  test  and  tlie 
criterion  measure  .78.  Tlie  K-K  20  reliability  of  the  Chinese  test  was  .96 
and  the  Russian  test  .95. 

m:VF.1.0I*>n: NT  of  THK  first  GEKTRATION  of  proficiency  tests  in  38  LAN’GUACK.S 
Language  specialists  at  The  Army  Language  Scliool  were  furnished  English 
translations  of  the  two  prototype  tests  mentioned  previously  and  were  tlien 
instructed  to  use  tliesc  trans  la  t ions , item  by  item,  in  developing  the  new 
tests.  Since  the  conditions  for  deviating  from  the  given  translations  were 
few  and  rattier  rigid,  the  resultant  new  tests  were  essentially  parallel  in 
many  ways  to  tlie  proti'types.  Unfortunately,  not  only  were  the  options  identi- 
cal, tiut  the  correct  option  remained  as  the  same  key  across  all  tests.  It  was 
subsequently  discovered  that  tlii.s  procedure  increased  the  chances  for  test 
compromise,  much  more,  than  usual.  The  reason  is  that  many  DLI  graduates  would 
cither  subsequently  reenroll  in  a different  language  course  or  take  a DLPT  to 
cstablisli  military  competency  in  a foreign  language  studied  elsewhere. 

Fur tlicrmorc , an  erroneous  assumption  was  more  than  likely  made  at  the  time 
that  since  all  the  tests  were  parallel  in  content  and  format,  they  would 
tliorefore  all  have  the  same  level  of  difficulty.  For  this  reason,  no  attempt 
was  made  to  standardize  raw  scores  and  the  same  skill-level  cut-off  scores 
were  used  across  all  languages.  This  Initial  assumption  was  unfortunate 
and  was  incorrect  as  later  validation  data  disproved  the  tlieory. 


DEVELOPMENT  OF  SECOND  GEKT.RATION  DEFENSK  LANGUAOl'  PROFICIENCY  TESTS  (DIPT  II) 


It  vas  considered  essential  to  correct  the  operational  situations  described 
above  and  to  rcj^lace  tlie  then  current  tests  which  were  outdated  and 
possibly  compromised  through  the  overuse  of  only  one  t si  form  per  language. 
Accordingly,  second  generation  DLPTs  arc  being  dtvtl>ped,  printed  and 
sys  t c ma  t i ca  1 1 y introduced  into  The  Di-fense  Language  Program  during  a three 
to  four  year  period.  The  new  second  generation  lists  have  the  following 
cha r a c t c r i s t i c s : 

a.  Although  they  are  identical  in  format  to  the  old  test,  the  content 
of  each  is  unique  and  indc))eiident  from  any  other.  Test  content  is  based 
strictly  on  a description  of  skill-level  objectives  for  each  language. 

h.  Current  plans  specify  at  least  three  alternate  forms  for  high- 
density  language.s  and  two  forms  for  low-enrollment  languages. 

c.  All  test  forms  will  be  subjected  to  item  analysis  and  form-equating 
procedures . 

d.  The  validation  of  each  test  will  be  based  on  the  collection  and 
analy.sis  of  field  data  concerning  job-per  formance  ratings. 

e.  Each  newly  developed  test  will  be  independently  normed  and  new  skill- 
level  cut-off  .scores  will  be  established. 

f.  Raw  scores  will  be  converted  into  standard  scores  to  achieve  mutual 
comparability  for  languages  which  have  varying  degrees  of  difficulty. 
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i’Koji;c:Ti;n  I'l.'i'OiK'i  (o^!  sait-i)iii.E  rOK  skcond  cknkration  dlpt.s 


DLI  R&D  PKIORITY  I (DLISDA,  P-0015) 


(CM) 

Chinesi'  (M) 

(U) 

Spanish  (LA) 

Arabic  (Modern  Standard)'" 

(Til) 

Tha  i 

(AK) 

Egypt ian 

(VN) 

Vietnamese  (N) 

(AP) 

Syr ian 

(VN) 

Vietnamese  (S) 

1 rail  i 

(RU) 

Russian 

Saudi 

Iht-si’  IjLP'i  ) I;i , Auilii*  lapc’s,  DI^l  IV.M  611-1,  OPSCAN  answer  sheets  are 
silieduleJ  J ur  d i s 1 1 i hnl  j on  world-wide  during  1st  and  2nd  Qtr,  FY  I'ly-'t. 

■•'•Two  Arahii  audio  tapi'.s  (AE , AP)  will  be  used  with  the  Arabic  test  booklet. 


1)1.1  R6.I)  PRIORITY  I (DLISDA,  P-0015-2) 


(FK) 

French 

(KP) 

Korea  n 

(CM) 

C(  riiviu 

(CM) 

Chinese-Mandarin  ' 

(.lA) 

• la pant  ;.i‘ 

(Simplified  ciiaracte 

rs) 

(YD) 

Ch inese-Amoy 

(originally  Priority 

111) 

iv.o  of  till  .1 

Dl.Pl  1 I have  been  completed.  'Ihe 

remaining  three  arc  scheduled  for 

1 1'.np  1 <■  t i . 'll  do 

rin;.'  3rd  Qtr,  FY  1974.  Publication 

and  wo  r 1 d - 

wide  distribution  is 

schi  liii  1 i d ilui 

ing  FV  197-'.. 

DLI  R6.D  PRIORITY  11  (DLISDA 

, P-0016) 

( TY) 

r.urmi  ;.i 

(PL) 

Po 1 ish 

(rc) 

t !t i ne si  -Ca n 1 oni  se 

(PQ) 

Portuguese  Brazilian 

(CX) 

Czi  ■ !: 

(TU) 

Turkish 

(mi; 

Hunga  r 1 r 

(CA) 

Cambiul  i.tn 

(IN) 

1 ndoiii  i -1  n 

(LC) 

Laotian 

Ihise  DLPl  11 

are  programed  !ir  eompletion  dining  3rd  and 

4th  Qtr,  FY  1974. 

DLI  RM)  PROIOTYPE  IKST  DEVELOPMENT  (PRIORITY 

111  AND  IV) 

(DI.ISDA  P-0055) 

Old) 

Hind  i 

(IIS) 

Hausa 

(AC) 

Ar.iha  r i c 

(HC) 

Ha i t i a n-Cr eo 1 e 

Iheso  arc  prototype  tests  designed  to  resolve  a deficiency  in  that  no  DLPTs  are 
currently  available.  Those  test  emphasize  the  evaluation  of  reading  proficiency 
in  the  target  language.  They  are  to  be  used  in  conjunction  v>)ith  a "linguistic 
d.iia  sheet"  in  order  to  assess  linguistic  background  factors  not  measured  by  the 
tests.  Scheduleil  completion  datc.s  are  3rd  and  4th  Qtr,  FY  1974.  ‘ 
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DLI  R&D  PRIORITY  III  (DLISUA,  P-0130-1 


(AB) 

Albanian 

(PT) 

Protuguese  (European) 

(BU) 

Bulgarian 

(RQ) 

Roma  nian 

(CM-M) 

Chinese -Main land 

(SC) 

Serbo-Croatian 

(OR) 

Greek 

(SR) 

Spanish  (Castilian) 

(JT) 

Italian 

(SW) 

Swahili 

(ML) 

Ma  lay 

(UK)* 

Ukranian  (deleted) 

(PF) 

Persian  (Farsi) 

R&D  work  is  underway;  completion  i.s  expected 

dur ing 

FY  1975. 

DLI  R&D  PRIORITY  IV 

(AA) 

Afr icaans 

(NR) 

Norwegian 

(OA) 

Danish 

(PG) 

Persian  (Afghan) 

(DU) 

Dutch 

(SL) 

S lovenian 

(F.T) 

F innish 

(SY) 

Swed ish 

(HE) 

Hebrew  (Mod) 

(UR) 

Urdu  \! 

Preliminary  R&D  work  is  underway;  due  to  the  lovjer  priority  a scheduled  completion 
date  ip  not, feasible  at  this  time.  \!  No  DLPTs  are  available  and  structured 
interviews  may  be  the  more  expedient  alternative. 
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Milton  H.  Maier 

paper  presented 

Military  Testing  Association 

San  Antonio,  Texas 

29  Oct  - 2 Nov  1973 

I 

Enlisted  Selection  and  Classification  Testing  j 

in  the  US  Army 

Introduction 

For  several  years  now  testing  has  been  in  a state  of  turmoil.  Pre- 
viously, tests  were  quietly  given  in  schools.  Industry,  and  the  armed 

services,  and  except  for  those  wlio  were  directly  Involved,  such  as  re-  j 

I 

searchers  and  personnel  managers,  no  one  really  paid  too  much  attention 
to  what  was  happening  in  tht  field.  In  the  1960's,  though,  testing,  along 
with  so  many  social  practices  and  customs,  was  called  into  question,  and 
today  testing  practices  are  regulated  by  law,  the  courts,  and  the  Equal 
Employment  Opportunities  Commission,  as  well  as  by  the  test  developers  and 
users.  Wliereas  in  former  times,  a new  test  might  be  of  local  interest  to 
the  immediate  users,  or  perhaps  to  the  professional  community  if  it  had  some 
unique  characteristics,  today  a new  test  is  subject  to  scrutiny  by  a good 

many  parties.  In  the  midst  of  this  concern,  the  new  Army  Classification  j 

Battery,  or  ACB,  was  born,  and  as  might  be  expected,  its  arrival  did  not  ' 

go  unnoticed.  In  my  paper,  I will  describe  briefly  what  the  new  ACB  is,  how 

it  is  used  operationally,  and  finally  give  some  comments  on  how  it  has  1 

j 

fared  since  its  introduction  on  1 May  of  this  year. 

What  is  the  new  ACB?  | 

The  new  ACB  arrived  on  the  scene  at  about  the  same  time  as  the  all  ^ 

I 

volunteer  Army,  and  it  is  admirably  suited  to  the  current  needs  for  i 

selecting  and  classifying  enlisted  personnel.  One  advantage  is  that  the 


I 

\ 
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new  battery  has  a broader  coverage  than  the  old  one.  The  two  primary 
changes  are  that  we  expanded  the  number  of  subtests  In  the  academic  or  general 
ability  domain,  and  that  we  Included  more  tests  of  Interests  and  attitudes 
or  measures  of  the  willingness  factor.  We  retained  the  old  standbys  of 
Arithmetic  Reasoning  and  Word  Knowledge,  and  we  added  measures  of  Science 
Knowledge  and  Mathematics  Knowledge  to  provide  more  precise  measurement  in 
this  critical  area  of  general  mental  abil*ity.  By  including  these  additional 
tests,  we  obtained  more  accurate  prediction  of  success  in  the  different  job 
areas.  (Chart  1) 

The  other  major  Improvement  is  that  we  added  more  interest  measures.  The 
earlier  ACB  contained  the  Classification  Inventory,  which  was  a self-description 
instrument  to  help  Identify  the  infantryman.  The  original  inventory  grow  out 

• Ws  did  not  ^ ^ ethics 

self-description  measures  in  the  ACB,  until  the  new  ACB  was  developed.  Now, 
we  still  have  a measure  of  combat  interest,  updated  during  the  Viet  Nam  con- 
flict, and  three  other  Interest  measures  in  electronics  and  mechanical  main- 
tenance, and  an  interest  in  attending  to  detail,  used  for  clerical-administra- 
tive jobs,  field  artillery,  and  for  equipment  operators,  such  as  drivers. 

Tlie  Interest  measures  add  to  the  prediction  of  success  in  relevant  job 
training,  in  addition  to  the  measures  of  general  ability  and  specific  measures 
of  information  in  a field,  such  as  electronics  information  and  automotive 
Information . 

The  information  tests  also  tap  a willingness  factor.  They  cover 
material  that  most  anyone  with  an  interest  and  aptitude  in  the  area  can 
pick  up.  Electronics  and  automobiles  are  common  to  our  culture,  and  a 
young  person  can  readily  become  expert  in  these  areas  If  he  has  the  Initia- 
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tlve  to  engage  in  relevant  activities.  And  as  part  of  our  rapid  social 
change,  young  women  no  longer  are  as  excluded  from  these  areas  as  in 
former  times. 

The  Trade  Information  subtest  covers  content  peculiar  to  the  skilled 
construction  trades. 

Finally,  we  have  two  perceptual  tests  that  do  not  involve  any  reading. 
These  are  the  familiar  patter'  analysis  or  spatial  ability,  and  a new  test, 
called  Attention  to  Detail,  which  requires  the  ability  to  discriminate  the 
letter  C embedded  in  a series  of  O's. 

The  tests  are  combined  to  provide  nine  aptitude  area  scores,  which  are 
used  as  prerequisites  for  job  training.  The  nine  areas  cover  the  full  range 
of  Army  jobs,  from  combat  and  field  artillery,  through  the  maintenance  areas  - 
electrom'rc^  general,  and  mechanical,  to  clerical  administrative  and  a set  of 
jobs  called  skilled  technical,  our  medics,  HP's,  military  intelligence,  and 
the  like.  We  have  two  other  job  areas  that  are  more  unique.  One  is  called 
surveillance  and  communications,  which  includes  radio  operators  and  target 
acquisition  jobs.  The  otfier  is  called  operators/food , which  includes 
missile  crewman,  cooks,  and  drivers.  (Chart  2) 

I should  say  that  all  of  these  job  groupings  are  empirically  based.  We 
did  not  force  the  jobs  to  come  out  in  any  special  way.  We  examined  the  pro- 
files of  validity  coefficients  both  visually  and  analytically  to  determine 
which  jobs  clustered  together,  and  the  grouping  shown  here  is  what  emerged. 

Looking  at  the  subtests  in  each  aptitude  area,  it  becomes  apparent 
that  Arithmetic  Reasoning,  AR,  is  omnipresent;  it  occurs  in  seven  of  the 
nine  areas,  plus  in  GT,  which  is  used  to  determine  eligibility  for  addi- 
tional testing  or  special  programs.  Another  fact  is  that  the  new  aptitude 
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areas  are  more  complex  than  the  old  ones.  These  contain  three  to  five  sub- 
tests each,  as  contrasted  to  only  two  in  the  old  aptitude  areas. 

The  result  of  putting  so  many  tests  in  each  composite  is  that  the 
new  aptitude  areas  are  more  accurate  predictors  of  success  in  job  training, 
which  is  good,  but  also  that  the  scores  are  more  highly  intercorrelated , 

which  at  first  seems  bad.  We  found,  however,  that  the  new  area  scores  do 
a much  better  job  of  sorting  out  the  manpower  into  appropriate  job  areas 
than  did  the  old  ones. 

We  analyzed  the  new  scores  by  subjecting  them  to  extensive  analyses 
through  simulating  scores  on  a computer  to  represent  random  samples  from  the 
mobilization  population.  We  assigned  the  simulated  "men"  to  job  areas  based 
on  both  the  new  and  old  aptitude  areas  scores.  We  kept  the  quota  restraints  that 
certain  percencages  had  to  go  into  each  job  area  which  reflected  the  real 
Army  quotas  at  the  time.  The  consistent  result  is  that  the  performance 
estimates  were  higher  under  the  new  set  of  scores  than  under  the  old.  The 
increase  in  predictive  accuracy  more  tlian  compensated  for  the  increased 
Intercorrelation . 

Another  gain  realized  from  the  new  aptitude  area  is  that  each  of  them 
contain  tests  that  require  the  ability  to  read.  In  former  days,  illiterate 
men  could  qualify  for  Army  service,  and  many  did.  Under  the  new  system, 
some  functional  illiterates  arc  still  getting  through,  that  is,  read  below 
the  fifth  grade  level,  but  the  number  has  been  reduced  to  an  estimated 
five  percent. 

Based  on  the  promise  of  more  accurate  measurement  of  job  potential, 
the  new  ACB  was  adopted  for  operational  use,  and  it  was  implemented  on 
1 May  1973.  Now  lot's  see  how  it  is  being  used. 
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Operational  use  of  the  new  ACB 

The  use  of  the  new  ACB  has  gone  through  two  phases  already  in  its 
short  life.  In  the  two  months  of  May  and  June,  a separate  AFQT  was 
administered  to  all  Army  male  applicants.  Beginning  1 July,  experimental 
mental  standards  were  adopted  by  the  Army  in  which  AFQT  was  dropped  as  a 
separate  test,  and  an  AFQT  score  is  obtained  from  the  ACB.  The  experimental 
mental  standards  were  also  changed  on  1 July  to  require  one  aptitude  area  score 
of  90  or  better  for  high  school  graduates  and  two  scores  of  90  or  better  for  non- 
graduates, in  addition  to  a percentile  score  of  10  or  better  on  the  AFQT  ob- 
tained from  the  ACB. 

The  new  aptitude  areas  made  the  experimental  standards  possible.  Since 
each  one  contains  some  measure  of  general  mental  ability,  plus  specific 
aptitude  measures,  a qualifying  score  of  90  is  more  indicative  of  ability  to 
succeed.  The  old  aptitude  areas  did  not  measure  endugh  general  mental  ability 
to  use  with  the  same  degree  of  confidence.  As  of  the  time  of  writing  no 
decision  has  been  made  about  which  standards  to  use  as  the  official  ones. 

The  ACB  is  now  given  at  the  time  of  application  of  enlistment,  which 
provides  the  recruiter,  counselor  and  applicant  with  the  best  possible 
information  about  the  applicants  mental  qualifications  and  the  job  areas 
in  which  he  is  best  qualified. 

The  tests  are  administered  at  AFEES  or  by  Mobile  Examining  Teams. 

METs,  who  take  the  tests  to  the  applicant.  The  entire  battery  requires 
about  three  hours  to  administer,  which  is  longer  than  the  previous  testing 
time  at  AFEES,  but  the  extra  time  is  well  spent.  The  new  recruit  actually 
spends  less  time  in  testing  than  before.  He  used  to  spend  up  to  eight 
hours  between  AFEES  and  reception  stations  taking  tests,  but  now  he 
spends  only  the  three  hours  at  AFEES,  plus  another  half-hour  taking  the 
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Auditory  Perception,  or  Radio  Code  test,  at  AFEES  if  he  is  applying  for  | 

training  requiring  radio  code,  or  at  reception  stations,  if  he  is  not  asking  j 

1. 

for  a coramitment  for  these  jobs.  | 

f 

The  new  ACB  is  also  given  to  women  applicants  at  AFEES  or  by  the  j 

Mobile  Examining  Teams.  An  AFWST,  i.e.  Armed  Forces  Womens  Selection  1 

j 

Test,  score  is  also  obtained  from  the  ACB.  No  additional  testing  of  women  j 

applicants  to  determine  qualification  is  required.  The  new  ACB  is  also  - 

being  readied  for  use  by  reserve  components  and  for  retesting  at  posts,  camps,  1 

and  stations  throughout  the  world.  Soon  the  entire  Army  will  be  switched 

f 

over  to  the  new  ACB.  The  Marines  also  are  using  the  new  ACB,  but  just  as 
with  the  Army,  the  transition  is  not  yet  complete. 

Evaluation  of  the  new  ACB. 

One  consideration  in  evaluating  the  new  ACB  w.as  its  effect  on  the  flow 
of  marginal  men,  or  those  with  low  general  mental  ability.  Prior  to  intro- 
duction of  the  new  battery,  our  research  had  shown  that  fewer  men  with  low 
AFQT  scores  would  obtain  qualifying  aptitude  area  scores  of  90  or  better. 

There  was  some  concern  tliat  tlie  effects  would  be  serious  enough  to  affect 
the  manpower  flow.  Our  argument  was  that  the  Army  has  little  difficulty 
in  recruiting  men  with  low  mental  ability,  and  the  experience  since 
1 May  has  borne  us  out.  In  July,  when  tlie  ceiling  on  the  number  of  men  with 
marginal  mental  ability  was  removed,  over  a third  of  the  accessions  were 
in  the  marginal  category.  In  August,  the  influx  subsided,  and  the  number 
was  more  acceptable  at  just  over  20  percent.  Army  policy  in  the  past  few 
years  has  been  to  hold  the  number  of  marginal  accessions  to  somewhat  under 
20  percent,  and  that  is  about  the  number  coming  in. 
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The  quality  of  new  accessions  In  the  past  was  measured  on  the  basis  of 
AFQT  scores,  which  provides  a single  index  of  general  tralnabillty . All 
men  regardless  of  aptitude  in  specific  job  areas,  were  categorized  on  the 
basis  of  their  general  tralnabillty.  With  the  old  aptitude  areas,  the 
AFQT  did  provide  the  best  measure  of  quality  and  the  procedure  was 
appropriate.  With  the  new  aptitude  areas,  categorizing  men  on  the 
basis  of  AFQT  is  no  longer  as  appropriate.  Since  the  new  scores  contain 
measures  of  general  ability,  the  number  of  qualifying  aptitude  area  scores 
at  or  above  specified  levels  provides  a more  accurate  description  of  the 
quality  of  the  accessions. 

We  have  been  working  with  Army  personnel  management  for  some  time  to 
develop  a new  basis  for  categorizing  quality.  Although  at  the  time  of 
writing  there  has  been  no  final  resolution  about  the  definitions  of  the 
categories,  it  is  clear  that  the  intent  is  to  broaden  the  basis  by  including 
all  the  aptitude  areas  in  developing  an  overall  quality  index.  The  basis 
that  is  finally  adopted  should  make  sense  from  all  points  of  view; 
some  considerations  are  that  it  should  have  a known  relationship  to  the 
mobilization  population,  it  should  provide  useful  information  to  personnel 
management  and  it  should  not  stigmatize  the  individual.  One  of  the  complaints 
about  the  AFQT  mental  categories  is  that  the  Cat  IV 's,  or  men  in  the  marginal 
category,  are  generally  assumed  to  be  the  misfits.  While  there  may  be  no 
way  to  avoid  stigmatizing  completing,  the  new  system  should  be  descriptive 
of  tralnabillty  in  specific  areas,  which  may  help  prevent  a general  labelling. 

Another  area  of  concern  about  tests  is  that  of  effectiveness  of  the 
tests  for  minority  group  personnel.  The  tests  should  be  equally  effective 
predictors  of  success  for  all  groups,  that  is,  blacks  and  whites  with  the 
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same  test  scores  should  have  the  same  expected  level  of  performance.  The 
new  ACB  is  equally  valid  for  blacks  and  whites,  and  blacks  and  whites  with 
the  same  test  score  have  the  same  predicted  level  of  performance  in  job 
training  courses. 

From  another  point  of  view,  the  question  is:  are  blacks  underrepresent;, 

in  the  Army  accessions  as  compared  to  the  population  as  a whole?  And  here 
the  answer  is  no.  In  July,  1973,  about  one-third  of  the  accessions  were 
black,  and  in  August,  20  percent.  These  humbers  compare  to  about  20  percent 
in  fiscal  year"' 1973,  and  about  13  percent  in  population  as  a whole.  The  net; 
ACB  thus  is  not  excluding  an  undue  proportion  of  blacks. 

The  matter  of  racial  bias  in  employment,  in  this  case  joining  the  Army, 
is  of  course  more  complicated  than  just  a matter  of  test  scores  or  even 
prediction  of  success  from  test  scores.  Test  scores  are  only  part  of  the 
system,  and  aujusting  only  the  scores,  without  also  adjusting  the  iuiticil 
job  training,  utilization  on  the  job,  and  advanced  training,  would  hardly 
be  a satisfactory  solution  for  either  the  individual  or  the  employer. 

Another  group  that  merits  special  consideration  is  women  applicants. 

The  new  ACB  is  used  to  determine  mental  qualification  of  Army  women  appli- 
cants. They  must  qualify  on  the  combination  of  Arithmetic  Reasoning  and 
Word  Knowledge  subtests,  which  provide  the  AFWST  score,  and  in  addition 
obtain  at  least  two  aptitude  area  scores  of  90  or  better.  The  Army  has 
had  little  difficulty  in  meeting  the  quotas  for  WAC's.  The  ACB,  however, 
was  standardized  on  an  all  male  sample,  with  both  blacks  and  whites 
Included,  but  no  women.  The  question  may  arise  whether  the  new  ACB  is  fair 
to  women.  The  answer  is  yes,  to  the  best  of  our  ability.  Based  on  large 
samples  of  males  and  females  in  high  schools,  the  population  means  on 
arithmetic  and  vocabulary,  which  constitute  the  new  AFWST,  are  about  the 

same  for  both  sexes.  As  for  the  aptitude  areas,  both  sexes  receive  the 
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same  job  training,  and  therefore  women  require  the  same  aptitude  as  men. 

If  the  prerequisite  for  job  training  is  100  or  average,  on  a particular 
aptitude  area,  women  as  well  as  men  need  to  meet  this  prerequisite  because 
students  of  both  sexes  need  to  master  the  same  knowledge  and  tasks.  The 
ACB  scores  for  women  are  considered  to  be  appropriate  measures  of  their 
potential  to  perform  satisfactorily  in  job  training. 

In  summary,  the  new  Army  Classification  Battery  has  been  in  use  for 
about  six  months.  It  is  administered  at  the  time  of  application  for  en- 
listment to  provide  information  about  the  applicant's  qualifications  in 
the  different  job  area.  The  evidence  is  that  it  is  not  keeping  out 
excessive  numbers  of  marginal  men,  blacks,  or  women.  Reports  received  from 
the  field  are  generally  favorable  in  that  it  is  an  efficient  and  effective 
method  to  assess  the  potential  of  applicants. 
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CONTENT  OF  NEW  ARMY  CLASSIFICATION  BATTERY 
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NEW  APTITUDE  AREA  COMPOSITES 
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THE  DEVELOPMENT  OF  THE  ENGLISH  COMPREHENSION 
LEVEL  SCREENING  TESTS  FOR  FOREIGN  STUDENTS 

BY:  Cortez  Parks 

The  English  Comprehension  Level  (ECL)  testing  system  is  the 
primary  quality  control  used  in  the  English  Language  Training 
Program  (ELTP)  conducted  by  the  Defense  Language  Institute,  English 
Language  Branch  (DLIEL)  at  Lackland  Air  Force  Base  and  at  numerous 
overseas  locations. 

It  is  a unified  testing  system  in  that  all  test  forms  are 
originated  and  controlled  by  one  agency.  The  construction, 
validation,  distribution,  administration,  and  control  of  the  various 
types  of  tests  prescribed  by  DLl  are  the  responsibility  of  the  Tests 
and  Measurement  Branch,  Development  Division,  English  Language 
Branch . 

Among  the  tests  produced  by  the  Tests  and  Measurement  Branch 
are  the  English  Comprehension  Level  Screening  Tests,  or  "ECL"  tests 
as  we  refer  to  them,  which  are  the  subject  of  this  paper.  The  ECL 
Screening  Tests  consist  of  general  English  tests,  usually  called  ECL 
tests  and  the  Specialized  English  Terminology  (SET)  tests. 

The  ECL  test  is  designed  to  measure  an  individual's  English 
listening  and  reading  comprehension  and  to  ascertain  his  capability 
to  acquire  knowledge  of  and  to  function  effectively  in  English 
language  training  and  work  situations. 

The  ECL  test  does  not  measure  speaking  or  writing  ability. 

These  skills  are  measured  subjectively  by  the  students'  instructors. 
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This  is  done  by  rating  the  speech  or  writing  productior.  by  the  use 
of  a speaking  and  writing  proficiency  level  code  key.  Correlation 
studies  made  at  DLTEL  of  the  Instructors  evaluation  compared  to  the 
ECL  indicate  that  the  subjective  evaluation  of  these  skills  has  a 
high  correlation  with  the  ECL  scores. 

ECL  tests,  as  well  as  the  SET  tests,  have  been  correlated  with 
language  performance  of  native  speakers  of  English  in  various 
specialized  fields  (Electronics,  Maintenance,  - - - - etc.). 

The  minimum  ECL  scores  and  cutoff  scores  of  the  SET  tests  have 
been  established  for  different  levels  of  training  in  the  various 
I specialties  of  the  Array,  Navy,  and  Air  Force.  These  scores  represent 

the  minimum  English  comprehension  level  necessary  for  foreign  students 
to  successfully  cope  with  English  instruction  in  tasks  involving 
varying  degrees  of  difficulty  and/or  danger. 

An  individual  cannot  normally  be  expected  to  perform  satisfactorily 
j and  efficiently  when  exposed  to  training  in  those  fields  with  an  ECL 

! score  below  the  minimum  requirements. 

I 

i A 100  ECL  score  Indicates  that  the  foreign  students  can  use 

standard  training  materials  without  any  difficulty. 

A 0-25  ECL  Includes  beginning  language  students  with  no  previous 
English  language  background  or  those  with  a weak  English  background. 

A 0-39  ECL  indicates  that  the  student  is  at  the  elementary  level. 
Language  instruction  given  at 'this  stage  consists  principally  of 

i 

carefully  selected  basic  vocabulary,  sentence  pattern  and  related 
pronuiiclatlon  drills.  No  formal  specialized  instructions  is  attempted. 

I 

1 
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A 40-59  ECL  indicates  that  the  student  is  at  the  intermediate 
I level. 

A 60-69  ECL  is  the  qualification  level  for  direct  entry  into 
j CONUS  or  3rd  country  apprentice  specialized  training. 

! A 70-79  ECL  is  the  required  qualification  level  for  direct  entry 

into  CONUS  or  3rd  country  in  most  basic  courses  (e.g.  primary  pilot 
training,  etc.). 

80  and  above  is  the  qualification  level  for  professional  career 
and  advanced  courses  (C  & GS  schools,  surgeons  and  advanced  medical 
courses,  pilot  transition,  advanced  flying  safety  officers,  demolitions, 
control  tower  operators,  etc.). 

I 

ECL  levels  are  not  percentage  scores  but  are  scores  converted  to  j 

established  qualification  levels;  i.e.  an  individual  with  a 70  ECL  j 

knows  much  more  than  twice  the  "amount"  of  English  than  one  with  a 35 
ECL,  and  the  progressive  proportions  between  45  and  90  ECL  are  not  the 
same  as  those  between  35  and  70.  The  ECL  spread  "value"  is  not  linear, 
but  is  more  like  a modified  logarithmic  curve. 

The  language  proficiency  levels  required  for  entry  into  the 
various  training  courses  are  set  by  the  services  responsible  for 
operating  the  training  facilities.  Applicants  failing  to  meet  the 
language  proficiency  level  requirements  for  direct  entry  are  programmed 
through  the  English  Language  Branch  for  additional  language  training. 

In  early  1962,  a number  of  the  service  schools  reported  that  a 
significant  number  of  students  were  entering  training  courses  who  were 
meeting  the  ECL  requirements  but  who  were  deficient  in  the  specialized 
terminology  peculiar  to  their  specialties.  This  was  especially  true  in 
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the  areas  of  electronics,  maintenance,  and  weather.  Tht'.  service 
schools  were  finding  it  necessary  to  give  these  students  extra- 
curricular instruction  in  technical  terminology.  At  a Tri-Service  ECL 
Test  Conference  at  Air  Training  Command  (ATC)  in  1964,  a decision  was 
made  to  add  SET  tests  to  the  ECL  screening  tests 

SET  tests  were  developed  in  five  specialized  areas;  electronics, 
maintenance,  supply,  weather,  and  medical.  Thus,  Series  6500  was  the 
first  series  of  the  ECL  screening  tests  that  consisted  of  ECL  and  SET 
tests . 

By  use  of  the  SET  tests,  Military  Assistance  and  Advisory  Groups 
(MAAGs)  were  alerted  to  the  necessity  of  programming  students  through 
the  DLIEL  Specialized  phase  prior  to  their  enrollment  in  CONUS  training 
courses . 

As  the  result  of  a language  training  survey  conducted  by  DLI  upon 
its  conception  in  1966,  it  was  decided  to  add  General  Army  and  General 
Navy  terminology  to  the  screening  tests.  These  two  additional  tests 
were  Incorporated  into  the  6700  Series  screening  tests.  Also,  several 
of  the  technical  training  schools  were  requested  to  submit  lists  of 
technical  terras  along  with  basic  course  materials  to  DLIEL  for  use  in 
the  development  of  SET  cests.  These  materials  are  updated  periodically. 

Tlie  7200  Series  SET  tests  now  in  use  and  the  7400  Series  which 
will  replace  the  7200  Series  in  July  1974,  consist  of  two  forms  each  of 
Electronics,  Maintenance,  Supply,  Weather,  Medical  Professional,  Medical 
Service,  General  Army,  and  General  Navy  tests  that  measure  the 
comprehension  of  technical  terminology  considered  essential  by  the 
respective  technical  training  schools. 
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The  ECL  test  consists  of  120  questions,  60%  of  which  are  on  tape 
and  test  the  aural  comprehension  of  the  student.  40%  of  the  questions 
test  reading  comprehension.  40%  of  all  objectives  tested  are  taken 
from  the  ALC  Elementary  and  60%  from  the  Intermediate  texts.  Items 
are  pres^ented  in  the  form  of  questions,  statements,  and  dialog  in  the 
listening  part  of  the  test  and  in  the  form  of  Incomplete  statements, 
underlined  objectives,  word  order,  and  complicated  sentences  in  the 
written  part.  75%  of  the  questions  consist  of  vocabulary  and  idioms 
and  25%  test  the  students'  abilities  to  select  the  proper  structure. 

All  items  are  of  the  multiple  choice  type  with  four  choices  of  answers, 
only  one  of  which  is  the  correct  answer.  The  standards  for  the 
construction  of  items  are  extremely  rigid. 

The  resulting  product  is  the  service  trial  edition  of  the 
screening  test  consisting  of  12  forms  of  ECL  tests  and  two  forms  of 
each  of  the  SET  tests. 

Validation  of  the  ECL  and  SET  tests  are  accomplished  by  pretesting 
the  service  trial  edition  with  the  foreign  students  at  the  various 
technical  schools.  Since  the  students  at  the  technical  schools  are 
all  in  the  upper  quadrant  of  language  proficiency,  the  ECL  tests  are 
pretested  also  among  students  in  the  lower  quadrant  at  overseas  schools. 

The  number  of  cases  used  for  experimental  testing  depends  upon 
the  type  of  statistical  analysis  required.  For  conducting  item 
analyses,  at  least  100  cases  were  used.  For  computing  correlation  or 
for  norming,  at  least  300  cases  were  obtained. 
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A great  amount  of  effort  is  being  expended  in  the  development 
of  the  ECL  screening  tests  in  order  to  obtain  the  highest  reliability 
and  validity  for  each  form  of  the  tests.  For  test  reliability,  both 
the  Internal  item  consistency  and  the  alternate-form  reliability  are 
carefully  checked.  For  test  validity,  we  check  the  content  validity, 
construct  validity,  and  predictive  validity.  The  reliability  index 
and  the  construct  validity  coefficient  of  each  of  the  different  forms 
have  been  kept  above  .90.  The  coefficients  of  the  predictive  validity 
have  shown  above  .65.  All  of  these  cutoffs  are  considered  high 
standards  in  test  validation.  For  an  example  of  statistical  data, 
please  refer  to  handout.  Table  I. 

We  have  also  correlated  the  ECL  tests  against  the  Test  of  English 
As  A Foreign  Language  (TOEFL)  tests,  the  University  of  Michigan 
language  proficiency  tests,  and  the  American  University  language  tests. 
Pretesting  has  also  been  conducted  at  San  Francisco  State  College,  the 
University  of  Texas,  Texas  A&M,  and  the  University  of  Minnesota.  The 
results  were  around  .80,  indicating  a high  correlation  with  their 
criterion  tests  of  English  proficiency. 

The  SET  tests  do  not  use  converted  standard  scores  as  do  the  ECL 
tests.  They  are  designed  to  use  percentage  scores  with  pass/fail 
cutoff  scores.  For  the  6500,  6700,  6900,  7200,  and  7400  SET  tests,  we 
have  used  minus  one  standard  deviation  as  the  cutoff  point  since  these 


are  proficiency  type  tests. 
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Tests  and  ECL  Tests 


AN  EXPEKIMENTAL,  MULTIMEDIA  CAREER  DEVELOPMENT 
COURSE  FOR  NEW  MENTAL  STANDARDS  AIRMEN 


George  P.  Scharf 

Air  Training  Command  - USAFSAAS,  Chanute 

This  paper  is  a report  on  an  experiment  to  determine  if 
modifying  the  Fire  Protection  Specialist  Career  Development 
Course  (CDC)  could  improve  the  CDC  as  a training  device  for 
low  mental  aptitude  airmen.  Modifications  to  the  regular 
CDC  included  simplifying  the  written  materials  and  adding 
18  hours  of  audio  supplementation.  The  experiment  used 
three  groups  of  New  Mental  Standards  (NMS]  Project  100,000 
airmen  enrolled  in  three  versions  of  the  CDC.  The  control 
group  took  the  regular  fire  protection  specialist  CDC.  The 
first  experimental  group  took  the  modified  CDC  with  audio 
supplementation.  The  second  experimental  group  took  the 
modified  CDC  without  any  audio  supplementation.  Criterion 
for  effectiveness  was  the  final  grade  performance  of  the 
personnel  in  the  three  groups.  Results  indicated  both 
experimental  groups  had  mean  final  grade  scores  significantly 
better  than  the  control  group.  However,  there  was  no 
significant  difference  between  the  mean  final  grade  scores 
of  the  two  experimental  groups.  Mean  course  completion  times 
for  the  three  groups  were  within  a week  of  one  another.  In 
this  experiment,  the  audio-supplemented  CDC  was  not  a cost- 
effective  method  of  course  presentation. 

In  recent  years  the  federal  government  has  been  quite  concerned  with 
improving  the  utilization  of  the  nation's  manpower  resources.  In  the 
late  60s  and  until  1971  there  was  considerable  pressure  on  the  military 
services  to  accept  and  train  personnel  with  low  mental  abilities.  One 
such  program,  Project  100,000,  began  in  1966.  This  program  called  for 
the  military  services  to  accept  up  to  100,000  enlisted  personnel  scoring 
between  the  10th  and  30th  percentiles  on  the  Armed  Forces  Qualification 
lest  (AFQTj , These  are  persons  in  the  Category  IV  mental  group. 

As  a part  of  Project  100,000  within  the  Air  Force,  a number  of  career 
fields  were  selected  for  implementing  the  use  of  large  numbers  of  marginal 
aptitude.  New  Mental  Standards  (NMS)  personnel.  While  NMS  airmen  were  a 
part  of  Project  100,000,  not  all  Project  100,000  airmen  were  NMS  people. 
The  definition  of  New  Mental  Standards  airmen  is,  "those  who  have  scores 
in  the  lower  half  of  the  Mental  Group  IV  range,"  One  of  the  career  fields 
selected  to  utilize  NMS  airmen  was  that  of  Fire  Protection  Specialist, 

AFSC  571X0. 

NMS  personnel  were  assigned  to  the  Fire  Protection  Specialist  Course 
at  Chanute  directly  from  Lackland.  However,  to  merit  this  assignment,  the 
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student  had  to  have  (or  achieve  before  leaving  Lackland)  a reading  level 
of  at  least  the  sixth  grade.  hTien  the  NMS  airmen  entered  training,  it 
was  found  that  most  of  these  personnel  with  low  mental  abilities  and  low 
reading  levels  could  complete  the  Fire  Protection  Course  if  given  adequate 
remedial  instruction,  counseling,  and  personal  guidance.  On  the  job,  however, 
where  structured  remedial  and  personal  help  were  not  as  readily  available, 
low  reading  ability  caused  great  difficulty  for  these  airmen  in  studying 
and  completing  their  Fire  Protection  Specialist  Career  Development  Course 
(CDC) . As  a result,  ATC  Headquarters  directed  that  an  experiment  be  set 
up  to  determine  if  simplifying  the  CDC  and  adding  audio  supplementation 
would  enable  students  to  complete  their  CDC  faster  and  with  better  grades. 

The  experimental  design  finally  selected  called  for  comparing  three 
equivalent  groups  of  NMS  airmen  who  had  successfully  graduated  from  the 
basic  Fire  Protection  Specialist  Course  at  the  School  of  Applied  Aerospace 
Sciences,  Chanute.  The  AFQT  scores  for  the  NMS  experimental  subjects 
ranged  from  20  down  to  10.  Upon  reaching  the  field  and  applying  for  their 
CDCs,  these  airmen  were  enrolled  in  one  of  three  different  versions  of  the 
Fire  Protection  Specialist  CDC.  The  students  in  all  three  groups  took  the 
same  pre-test  and  final  or  post- test.  The  three  CDC  versions  covered  the 
same  basic  information,  but  their  modes  of  presentation  differed  as  follows: 

CDC  57151A.  Airmen  who  took  this  course  were  the  control  group.  This 
is  the  conventional  Fire  Protection  Specialist  CDC. 

CDC  57100.  This  was  an  experimental,  multimedia  CDC  which  used  short 
sentences”  large,  single-column-per-page  print,  contained  more  pictorial 
materials  than  the  regular  CDC,  and  used  cassette  tape  recordings  to 
summarize,  emphasize,  and  express  in  different  terms  the  information  read 
by  the  student.  No  new  or  different  information  was  included  in  the 
recordings.  IvTiile  detailed  instructions  on  using  the  tape  player  with  the 
written  material  are  a part  of  the  CDC  text,  the  procedure  was  somewhat 
as  follows: 

(1)  After  reading  an  instruction  sheet  on  the  tape  player  and 
directions  at  the  start  of  the  first  CDC  Volume,  the  student  studied  the 
first  part  of  his  CDC.  After  a few  paragraphs,  he  was  instructed  to  put 
a particular  cassette  into  the  tape  player  and  turn  it  on. 

(2)  The  tape  recording  referenced  the  passages  the  student  had 
just  read.  Speaking  in  turn,  a group  of  up  to  four  male  voices  then 
discussed,  summarized,  restated,  or  reviewed  the  information  in  the 
referenced  paragraphs.  The  student  was  then  directed  by  a recorded  voice 
to  turn  the  player  off.  He  then  returned  to  the  written  text  until  he 
reached  the  printed  instruction  to  again  turn  on  the  tape  player.  This 
procedure  was  followed  throughout  the  entire  five  volumes  of  the  CDC  for 
a total  of  approximately  18  hours  of  audio  review  of  what  the  student 
had  read . 
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(3j  Volume  introductions,  chapter  introductions,  and  chapter 
sununaries  were  recorded  witli  a female  voice  to  give  more  variety  to  the 
tapes . 

CPC  57100B.  I'his  was  the  same,  less  verbal,  more  pictorial 
CDC  57100  without  the  tape  recordings  and  with  all  references  to  using 
a tape  player  removed  from  the  wTitten  text. 

Procedures 

Starting  in  August  1971,  a graduation  roster  of  each  class  in  the 
Fire  Protection  Specialist  Course  was  furnished  the  Training  Research 
Applications  Branch.  The  first  entries  into  the  CDC  experimental  program 
graduated  from  the  Chanute  course  on  7 September  1971.  When  an  NTIS  airman 
graduated,  his  name  and  social  security  account  number  were  furnished  to 
the  Extension  Course  Institute,  Educational  Systems  Branch  (ECl/EPXV) , 
at  Gunter  AFB  in  Alabama.  These  names  were  then  put  on  an  identifier  list 
at  ECI  as  eligible  for  the  CDC  experimental  program. 

a.  The  number  and  flow  of  NMS  personnel  available  for  this 
experiment  never  became  as  great  as  the  originally  projected  16  per  month. 

One  reason  for  tliis  was  that  Project  100,000  unofficially  decreased  to 
Project  50,000  some  months  before  the  experiment  began.  As  a result,  the 
number  of  NMS  students  entering  into  the  Fire  Protection  Specialist  Course 
decreased  by  more  than  half.  I'hus , by  the  time  the  CDC  course  materials 
were  prepared,  there  was  a shortage  of  eligible  participants  for  the 
experiment . 

b.  Effective  1 April  1972,  Change  4 to  AFM  53-3,  Enlistment  in  tlie 
Regular  Air  Force,  cut  off  the  NMS  student  flow  entirely.  Under  this  change. 
Air  Force  enlistees  must  have  an  AFQT  of  at  least  21,  and  if  the  AFQT  is 
between  21  and  30  the  enlistee  must  be  at  least  a high  school  graduate. 

This  change  in  enlistment  requirements  stopped  entry  into  the  Air  Force 
(and  tlms  eventuall)’  into  this  experiment)  of  NTIS  airmen.  As  a result, 
the  number  of  cases  analyzed  in  the  study  is  as  follows:  regular  CDC  -- 

24,  experimental  group  1 using  the  tape  recorder  --  25,  experimental  group  2 
using  only  the  rewTitten  course  materials  --  19. 

As  applications  for  the  Fire  Protection  Specialist  CDC  were  received 
by  ECI,  the  NMS  identified  personnel  were  assigned  to  one  of  the  three 
groups  described  above,  i.e.,  CDC  57151A  (control  group),  CDC  57100 
(multimedia  experimental  group),  CDC  57100B  (second  experimental  group). 

Before  being  sent  liis  course  materials,  each  student  was  administered 
a 100-question  pre-test  by  his  OJT  supervisor.  Ivlien  ECI  received  the 
pre-test,  the  written  CDC  materials  for  tlie  course  to  which  he  was  assigned 
were  sent  to  the  student.  The  tape  players  for  the  multimedia  CDC  57100 
were  sent  from  Chanute.  Students  in  all  three  CDCs  took  the  same  pre-test 
and  final  test. 
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The  measures  of  effectiveness  of  the  two  experimental  CDC  groups  was 
an  analysis  of  their  final  course  grade  performance  in  comparison  to  the 
control  CDC  group. 

All  calculations  were  performed  using  statistical  packages  available 
in  the  PLATO  IV  computer-based  teaching  system.  There  are  presently  four 
PLATO  IV  terminals  at  Chanute  connected  to  a central  computer  located  at 
the  Urbana  campus  of  the  University  of  Illinois.  The  mathematical  formulas 
necessary  for  any  analysis,  including  those  involving  unequal  n's,  are 
included  in  the  analysis  programs  used.  The  person  making  the  analysis  is 
required  only  to  select  the  type  of  analysis  desired  and  input  the  raw  data. 
Answers  appear  on  the  screen  immediately.  Thus,  it  was  possible  to  examine 
a great  amount  of  raw  data  concerning  the  subjects  of  the  study  and  receive 
back  an  almost  instantaneous  readout  of  the  analysis  selected. 

Findings 

Overall  results  obtained  from  this  study  are  summarized  in  Table  1. 


Table  1 
CDC 

CDC 

CDC 

57151A 

57100 

57100B 

n=24 

n=25 

n=19 

AFQT  mean 

15.35 

15.08 

15.05 

AFQT  standard  deviation 

3.41 

3.55 

3.40 

Resident  course  grade 
mean 

83.75 

83.88 

84.61 

Resident  cour-se  grade 
standard  deviation 

3.43 

3.69 

3.68 

Pre-test  grade  mean 

56.58 

57.40 

61.95 

Pre-test  grade  standard 
deviation 

10.58 

9.57 

10.78 

Post-test  grade  mean 

66.75 

72.80 

76.00 

Post-test  standard 
deviation 

7.71 

10.73 

11.07 

Net-mean  grade  score 
increase 

10.17 

15.40 

14.05 

Net-mean  grade  score 
increase  standard 
deviation 

9.59 

8.72 

8.98 

Completion  time  mean 

205.00  days 

211.50  days 

205.32  days 

Completion  time  standard 
deviation 

76.45  days 

50. 26  days 

31.64  days 
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To  establish  iliat  the  three  groups  were  equivalent  in  ability,  a 
one-way  analysis  of  variance  was  conducted  on  the  means  of  the  students' 
AFQTs,  final  grades  from  the  resident  Fire  Protection  Specialist  Course, 
and  their  CDC  pre-test  scores.  Table  2 summarizes  the  one-way  analysis 
of  variance  on  each  factor  among  the  three  groups. 


Table  2 


Source  of  V'ariation 

F 

d.f. 

E 

AFQT 

0.045 

2,65  = 

0.9558 

Resident  Course 
Final  Grade 

0.341 

2,65  = 

0.7122 

CDC  Pre-test 
Grade 

1 . 628 

2,65  = 

0.2042 

As  can  be  seen,  the  F ratios  for  each  of  these  three  student  variables 
were  all  non-significant,  thus  indicating  comparable  groups. 

In  order  to  test  tlie  effectiveness  of  the  instructional  methods,  a 
one-wa)'  analysis  of  variance  was  conducted  on  the  means  of  the  post-test 
CDC  (final  grade)  scores  achieved  by  the  three  groups.  Table  3 presents 
this  analysis  of  variance. 


Table  5 

ouiuce  of  I'ariation  i-  d . f . p 


Post-test  score  4.9bb  2,65  = 0.0098 


Inspection  of  fable  5 shows  that  the  post-test  score  F ratio  was  4.966; 
this  is  a significant  value  indicating  the  post-test  performance  of  the 
three  groups  was  not  comparable. 

To  compare  the  differences  of  the  tliree  groups  on  the  post-test 
performance  factor,  a series  of  t tests  were  run.  Their  results  are 
summarized  in  fable  4. 


56 


Table  4 


II 

Mean  post-test  score 
of  Ex  Gp  1 [revised 
text,  audio  supplement) 

I 

72.80 

Mean  post- test  score 
of  Ex  Gp  2 (revised 
text  only) 

76.  00 

Mean  post-test  score 
of  Ex  Gp  2 (revised 
text  only) 

76.00 


Mean  post-test  score 
of  control  group 


66.75 

Mean  post-test  score 
of  control  group 


66.75 

Mean  post-test  score 
of  Ex  Gp  1 (revised 
text,  audio  supplement) 

72.80 


t d.f.  £ 


2.258  47.0  .06 


3.227  41.0  .05 


.967  42.0  .44 


The  inferences  from  Table  4 are  that  both  the  audio  supplemented  CDC 
and  the  revised  text  only  CDC  yielded  significantly  higher  grades  than  the 
regular  CDC.  However,  there  was  no  significant  difference  in  the  final 
course  grades  of  the  two  experimental  groups. 

To  establish  the  equivalency  of  the  completion  times,  a one-way 
analysis  of  variance  was  made  on  their  means.  This  analysis  is  shown  in 
Table  5. 


Tabic  5 


Source  of  Variation 


F d.f. 


P 


Completion  time  of 
CDC  in  days 


0.064  2,65  = .9383 


The  indication  of  this  analysis  is  that  the  CDC  completion  times  for 
the  three  groups  were  not  significantly  different. 
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Discussion 


In  tliis  study,  Dotli  experimental  ClK!  groups  had  sta_istically  significant 
better  learning  scores  than  the  regular  CDC  students.  However,  there  was 
no  significant  difference  in  the  learning  scores  of  the  two  experimental 
groups.  IVliile  not  statistically  sigrificant,  the  experimental  group  without 
the  audio  supplementation  had  a mean  final  grade  that  was  3.2  points 
higher  than  the  experimental  group  using  audio  supplementation.  At  the 
same  time,  the  net -mean  grade  score  increase  of  these  two  groups  was  1.78 
higher  for  tliose  using  audio  supplementation.  Both  experimental  groups 
had  greater  net-mean  grade  score  increases  than  the  control  group. 

IVhile  costs  of  a program  sucli  as  this  are  a rather  elusive  thing  and 
hard  to  pinpoint,  it  would  appear  that  the  production,  implementation,  and 
administration  of  the  audio-supplemented  CDC  was  between  three  and  four 
times  the  cost  of  the  regular  CDC.  This  factor  remains  about  the  same 
when  projected  over  a much  larger  number  of  students  than  participated 
in  tliis  experiment.  For  all  three  CDCs  the  production  of  the  written 
materials  is  equivalent.  In  addition,  the  larger  the  printing  of  a CIX.’, 
the  less  cost  per  student.  However,  the  premise  that  the  larger  the  audio 
group  the  less  the  cost  per  student  does  not  appear  valid.  Ivliile  there 
might  be  some  slight  savings  in  buying  a large  number  of  cassettes  and 
tape  players,  t!ie  loss  rate  plus  maintenance  and  replacement  costs  of 
these  items  would  nullify  any  projected  reductions  in  cost  per  student 
based  on  a large  number  of  users.  Consequently,  it  does  not  appear  cost 
effective  for  widespread  adoption  of  an  audio- supplemented  CDC  program 
when  students  did  no  better  (on  the  average)  than  the  students  who 
received  only  the  revised  text.  Sellman,  1970,  in  a similar  experiment 
used  subjects  of  low,  medium,  and  high  aptitude  and  found  that  it  was 
low-aptitude  personnel  who  showed  the  greatest  learning  gains  when  audio 
supplementation  was  used.  Since,  at  this  writing,  these  low-aptitude 
personnel  are  not  accepted  b)'  the  Air  Force,  the  use  of  audio  supplementa- 
tion to  CDCs  would  appear  to  have  even  less  potential  as  a cost  effective 
teaching  device  in  the  CDC  program. 

OF  the  31  NMS  airmen  assigned  to  the  57100  course,  four  were  dis- 
charged before  completing  training,  one  desserted,  and  one  is  in  confinement. 
This  leaves  the  number  of  N'MS  subjects  who  completed  training  at  25.  Nine 
tape  players  were  lost  or  stolen. 

At  the  end  of  CDC  57100,  students  were  asked  to  comment  on  the  use  of 
tlie  tape  pla_\-er  ami  its  effect  on  jjK^course.  In  general  , the  reaction  of 
students  was  favorable,  but  not  overwhelmingly  in  support  of  the  audio 
tapes,  ihe  student's  supervisor  was  also  asked  to  comment  on  the  use  of 
the  tape  player.  Here  too,  the  general  reaction  to  the  tape  player  was 
favorable  with  one  or  two  of  the  supervisors  very  strong  in  their  support 
of  the  idea  and  one  or  two  as  strongly  opposed.  Fifteen  of  the  supervisors 
seemed  to  feel  the  tape  helped  the  student  to  finish  faster  than  normal. 
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However,  the  mean  time  for  those  using  the  audio  tapes  was  a week  longer 
than  those  using  the  other  versions  of  the  CDC.  The  difference  in  completion 
times  for  the  three  groups  was  not  significant. 

Conclusions 

As  a result  of  the  analysis  of  the  data  evolved  from  this  study,  the 
following  conclusions  are  hereby  stated. 

The  two  experimental  groups  had  better  final  CDC  grades  than  did  the 
control  group. 

The  differences  in  grades  of  the  two  experimental  groups  was  not 
statistically  significant.  The  audio-supplemented  experimental  group 
had  a net-mean  increase  of  their  final  grade  over  their  pre-test  grade 
1.78  points  more  than  the  non-audio  supplemented  group;  but  the  non-audio- 
supplemented  group  had  a mean  final  grade  3.2  points  higher  than  the 
audio -supplemented  group. 

Any  higher  grades  of  the  experimental  groups  can  be  attributed  to 
the  revised  text  as  much  as  to  any  audio  supplementation  as  all  three 
groups  were  shown  as  equivalent  in  mental  ability  and  pre-course  knowledge. 

Mean  completion  times  for  the  three  groups  were  within  a week  of  one 
another. 

Audio  supplementation  was  not  a cost-effective  means  of  instruction 
in  this  experiment. 
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Optimal  Utilization  of  On-the-Job-Training 
and  Technical  Training  School 

by 

Captain  AI'mi  D.  Dunham 


This  paper  discusses  work  planned  and  completed  during  the  AF 
Human  Resources  Laboratory’s  effort  to  develop  techniques  for 
optimizing  the  use  of  On-the-.Iob  Training  (OJT)  and  Technical 
Training  School  (TTS)  as  methods  of  initial  upgrading  for  non-prior 
service  personnel. 

I.  INTRODUCTION 


The  author's  previous  MTA  paper  (1972)  discussed  criteria  relevant 
to  selecting  optimal  OJT/'TTS  mixes  and  then  described  the  results  of 
work  which  established  the  feasibility  of  obtaining  OJT  cost  data. 

Significant  improvements  in  the  OJT  costing  methodology  have  been 
obtained  in  the  year  since  the  first  report  and  progress  has  been  made 
in  the  application  of  a measure  of  one  other  relevant  criterion,  the 
quality  of  tlie  training  methods.  This  progress  is  discussed  in  the 
following  text. 

II.  REFINEMENT  OF  CJT  COSTING  METHODOLOGY 

The  original  OJT  costing  methodology^  identified  the  following  cost 
factors; 


Table  1 

OJT  Cost  Factors 

Student  Time 
Instructor  Time 
Delayed  Entry  into  Training 
Records  Management 
Remedial  Training 
Equipment  and  Materials 


1 Dunliam,  1972 
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Data  concerning  these  cost  factors  were  collected  by  survey  for  the 
Communications  Center  Operations  specialty  for  OJT  to  the  3 (semi- 
skilled) level  for  non-prior  service  airmen.  These  surveys  asked 
supervisors  in  this  job  specialty  to  provide  detailed  data  on  several 
items,  including  time  spent  learning  a number  of  skills  specific  to 
Communications  Center  Operations.  The  Supervisors  had  a wide  range 
of  experience  with  OJT.  The  resulting  OJT  estimate  was  considerably 
less  than  the  cost  of  TTS  for  the  corresponding  training  course. 

The  most  important  aspect  of  this  research  was  that  it  established 
the  feasibility  of  obtaining  cost  estimates  for  USAF  OJT  through  the 
use  of  a survey  technique. 

One  undesirable  result  was  the  high  response  variance  encountered 
for  many  of  the  crucial  questions.  This  resulted  in  relatively  wide 
confidence  limits  for  the  OJT  cost  estimate,  but  not  wide  enough  to 
cast  doubt  on  the  cost  comparison.  Another  undesirable  characteristic 
was  that  a large  section  of  the  survey  instrument  was  specific  to 
Communications  Center  Operations.  A survey  instrument  independent 
of  the  job  performed  would  require  less  resources  for  survey  administra- 
tion and  data  file  maintenance. 

m.  THE  DEVELOPMENT  AND  EVALUATION  OF  ALTERNATIVE 

METHODOLOGIES  FOR  ESTIMATING  THE  COST  OF  AIRFORCE  OJT 


In  June  of  1972  Manpower  and  Personnel  Systems  Division  of  the  AF 
Human  R,  sources  Laboratory  let  contract  #F41609-72-C-0048,  designed 
to  improve  upon  the  original  OJT  costing  methodology.  During  the  first 
phase  of  this  contract  the  contractor  used  the  original  survey  approach 
and  two  alternative  survey  techniques  to  simultaneously  collect  cost  data 
concerning  OJT  to  the  3-level  for  USAF  Administrative  Specialists. 

One  of  the  two  new  alternative  survey  techniques  asked  supervisors 
to  estimate  information  for  eleven  variables  for  the  previous  week, 
including  time  spent  on  OJT  by  instructors  and  trainees.  The  other 
alternative  survey  technique  required  supervisors  to  record  time  spent 
by  instructors  and  supervisors  on  a daily  basis  for  one  week. 

Under  ideal  circumstances  each  survey  technique  would  be  used  on 
a unique  set  of  supervisors.  The  time-phasing  of  the  contract  and  an 
unexpected  reduction  in  the  number  of  Administrative  Specialists  entered 
in  OJT  forced  administration  of  all  three  survey  instruments  to  tlie  same 
set  of  supervisors.  The  three  distinct  survey  techniques  were  preserved 
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in  the  instrument,  but  most  of  the  redundant  questions  were  asked  only 
once  to  minimize  the  length  of  the  survey. 

Of  the  295  surveys  sent  to  supervisors  at  88  Air  Force  bases,  207 
were  returned.  All  responses  were  manually  edited  and  keypunched 
for  transferal  of  the  survey  data  to  magnetic  tape. 

The  questionnaire's  length  might  have  been  responsible  for  a reduced 
response  rate  toward  the  end  of  the  instrument,  but  response  rates  for 
all  three  survey  approaches  were  sufficient  to  enable  comparison  of  the 
alternative  techniques. 

A detailed  report  of  the  survey  data  and  an  in-depth  discussion  of 
their  implications  will  be  forthcoming  soon  in  an  A F Human  Resources 
Laboratory  technical  report.  For  brevity  and  to  minimize  duplication 
Table  2 describes  item  content  and  mean  responses  without  discussion 
of  their  implications.  The  reader  is  cautioned  against  inferring  strong 
conclusions  from  this  limited  display  of  data. 

A cost  estimate  for  each  OJT  cost  factor  is  exhibited  in  Table  3.  The 
cost  models  used  to  compute  these  estimates  were  selected  from  several 
cost  models  based  on  various  combinations  of  the  three  survey  approaches. 
Again,  a more  rigorous  discussion  will  soon  appear  in  AFHRL  publication. 
The  OJT  cost  per  trainee  of  $1545  can  be  compared  to  the  marginal  cost 
per  trainee  of  course  3ABR  70230,  $2271.^  There  are  criteria  other 
than  costs  which  are  equally  important,  viz.  , training  capacities  and 
quality  of  the  training.  Thus,  conclusions  regarding  optimal  OJT/TTS 
mixes  for  the  Administrative  Specialty  based  only  on  data  in  this  report 
would  be  reached  without  considering  other  important  criteria.  These 
criteria  are  discussed  in  a Technical  Report  in  review  at  AFHRL. 


T TTif  cost  of  Tech  scHbor  was  obtained  from  RAND  Corporation  data. 


Table  2 


Mean  Responses  for  Admin  Specialist  OJT  Survey 
Responses  Based  on  Supervisors’  Total  Experience 

Item  Content  Mean  Response 


Persons  in  Section 

Days  Delay  in  start  of  3-level  OJT 

Days  Delay  in  start  of  5-level  OJT 

Weeks  to  3-level  Proficiency 

Weeks  from  Proficiency  to  Actual 

Award  of  3-level 

Number  of  3-level  Trainees 

Number  of  5-level  Trainees 

Additional  Training  Capacity 

Operate  with  fewer  personnel  if  no  OJT 

Are  Trainees  Productive? 

OJT  Superior  to  TTS  Grad 
TTS  Superior  to  OJT 

No  Difference  between  OJT  and  TTS  Grads. 
Percent  Failing  Apprentice  Knowledge  Test 
Weeks  Remedial  Training 
Trainee  Remedial  Hours/Week 
Instructor  Remedial  Hours/Week 
Training  Record  Keeping  Hours/Week 


7.27 
8.60 
8.44 
13.  30 

3.31 
1.34 
1.06 
1.11„ 
29% 
82%  ^ 
22%  ^ 
32% 
46%  3 
25.64 
3.60 
9.07 

5.28 
1.94 


Skill  Area 


4 


Wks.  to  Prof. 


Instructor  Hrs/Wk.  Trainee  Hrs/Wk. 


Career 

2.86 

2.  94 

7.  78 

Security 

2.05 

1.  78 

3.89 

Supervision 

4.80 

3.  22 

6.90 

Equipment 

3.78 

3.31 

9.14 

Publications 

4.  51 

3.59 

7.83 

Forms 

3.64 

2.65 

5.43 

Communications 

4.99 

3.27 

8.28 

Documentation 

5.02 

3.04 

6.92 

Library 

2.70 

1.  16 

4.02 

Postal 

3.  10 

2.07 

5.04 

15  Per  cent  of  total  valid  responses  that  responded  affirmatively. 

4 Corresponds  to  the  Job  Proficiency  Guide  skill  breakdown  for  this 
speciality. 
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Table  2 (Cont’d) 


Week 


Productive  Trainee  Hours  (Mean  Response) 


1 

H.87 

4 

17.92 

8 

23.44 

12 

27.25 

16 

30.78 

20 

33.21 

Responses  Based  on  Supervisor's  Experience  Previous  Week: 


Item  Content  Mean  Response 


Week  of  Training  for  Trainee  10.88 

Percent  of  Training  Completed  56.21 

Percent  of  Skills  known  prior  to  OJT  10.  93 

Days  of  delay  in  start  of  3- level  OJT  9.  16 

Trainee  Hours  OJT  9.07 

Instruccor  Hours  10.  88 

Record  Keeping  Hours  1.  88 

Days  of  delay  in  start  of  5-level  OJT  6.  71 

Percent  of  skills  known  at  time  of  arrival  at  unit 

(TTS  Grads)  29.  57 

Additional  weeks  to  3-level  Proficiency  5.05 


Responses  Based  on  Supervisor's  Record-Keeping  for  one  Week; 


Item  Content 


Mean  Response 


Number  of  Trainees  1.  3 

Trainee  hours  instruction  10.  61 

Trainee  Hours  productive  24.  73 

Instructor  hour  s/trainee  8.80 
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Table  3 


Summary  of  OJT  Costs 


Cost  Factor 


Cost/Trainee 


Trainee  Time 
Instructor  Time 
Delayed  Entry 
Records  Management 
Remedial  Training 
Equipment  and  Material 


$ 570.18 
591.35 
196.05 
137.70 
40.03 
10.18 


$1545.49 


Tables  4 through  6 summarize  the  more  obvious  conclusions  concerning 
the  three  survey  approaches  used.  When  sample  size  is  no  problem  and 
when  information  specific  to  the  skill  groups  is  not  needed,  the  survey 
approach  used  should  be  that  of  Table  6.  When  sample  size  is  a problem, 
the  methodology  of  Table  5 would  suffer  from  the  same  problem  as  that 
of  Table  6,  so  that  a form  of  the  original  survey  approach  appears  more 
advantageous.  Most  limitations  on  sample  size  can  be  overcome  by 
extending  the  time  span  of  the  survey. 

Selection  of  a survey  approach  thus  involves  trade-offs.  The  Phase  I 
report  of  this  contract  recommends  use  of  the  general  survey  approach 
of  Table  6 while  retaining  some  of  the  questions  used  in  the  original 
methodology.  This  synthesis  of  the  survey  approaches  was  used  in 
Phase  II  of  the  contract  and  appears  as  Appendix  A of  this  paper. 
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Table  4 


Original  Survey  Approach  - Responses  Based  on  Supervisor’s  Total 
~~  Experien'ce 


Advantages  (compared  to  alternative  methodologies): 

a.  Larger  sample  at  any  point  in  time 

b.  Breaks  out  information  by  skill  groups  specific  to  speciality 

or  job  performed 

c.  Can  be  completed  in  one  sitting  - minimizing  turnaround 

and  survey  control  workload 


Disadvantages: 

a.  Speciality  - dependent 

1.  makes  aggregation  of  OJT  cost  data  across  specialities 

more  complex 

2.  requires  that  a different  survey  form  be  developed  for 

each  speciality 

3.  requires  multiple  formats  for  maintenance  of  OJT 

cost  data. 

b.  High  response  variation,  resulting  in  relatively  high  variance 
around  OJT  cost  estimate. 
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Table  5 


Alternative  Survey  Approach  - Responses  Based  on  Supervisor’s 
Experience  During  Previous  Week  “ 

Advantages: 

a.  Can  be  completed  in  one  sitting; 

b.  Speciality-Independent; 

c.  Variance  reduced  somewhat  compared  to  original  methodology; 

d.  Recall  required  is  for  more  recent  experience. 

Disadvantages: 

a.  Provides  no  data  on  costs  related  to  specific  skill  groups; 

b.  Requires  presence  of  OJT  trainee  during  previous  week, 

which  may  result  in  a reduced  sample  size  for  any  point  in  time. 
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Table  6 


Alternative  Survey  Approach  - Responses  Based  on  Record-Keeping 

for  One  Week 


Advantages: 

a.  Generally  reduced  response  variance  compared  to  other  survey 

approaches; 

b.  Speciality  - independent; 

c.  Recall  involved  limited  to  one  day  of  work. 

Disadvantages: 

a.  May  have  reduced  sample  size  due  to  need  for  currently 

enrolled  trainee; 

b.  Turnaround  time  is  longer; 

1.  may  result  in  reduced  sample  size  due  to  increase  in 

time  in  the  field; 

2.  will  require  more  record  keeping  by  survey  control. 


Phase  II  of  the  O.IT  costing  contract  is  nearly  completed.  During 
this  phase  the  contractor  utilized  the  survey  technique  developed  in 
Phase  I to  collect  cost  data  for  OJT  to  the  3-level  for  the  following 
specialities:  Fire  Protection,  pavements  Maintenance,  Fuels, 

Materiel  Facilities,  and  look. 

The  results  of  this  phase  will  soon  be  ready  for  publication.  Persons 
interested  in  these  results  should  be  able  to  obtain  information  from 
Manpower  and  Personnel  Systems  Division  of  the  AFHRL  in  the  near 
future. 

IV.  QUALITY  OF  THE  TRAINING  METHODS 


The  OJT  costing  work  described  above  shows  that  the  Air  Force 
can  and  is  obtaining  OJT  cost  estimates  compatible  for  comparison  with 
TTS  cost  estimates.  One  assumption  required  for  a straightforward 
comparison  is  that  these  two  methods  of  training  provide  trained 
personnel  of  equal  quality  in  terms  of  their  productivity  on  the  job. 

Cost  estimates  which  included  'quality'  in  some  manner  would  be 
tenuous  figures  based  on  unsupported  assumptions  because  measures 
of  quality  useful  for  this  purpose  do  not  exist.  Hence,  cost  estimation 
and  quality  comparisons  are  treated  separately. 

The  following  discussion  of  methods  of  quality  comparison  is  a 
summary  of  some  work  completed  at  Manpower  and  Personnel  Systems 
Division,  AFHRL.  A thorough  analysis  of  the  'quality'  problem  is  in 
preparation  and  should  appear  as  an  AFHRL  technical  report  within  a 
year. 

Tlie  problem  of  comparing  the  quality  of  training  methods  in  the 
military  context  deserves  a thorough  theoretical  discussion  of  the 
impediments  faced  by  researchers.  It  was  thought  expedient  to  publish 
some  interesting  data  in  this  article  with  only  a short  discussion  of 
problem  traits  to  allow  interested  persons  an  earlier  glimpse  of  the 
results. 

Previous  research  concerning  this  subject  is  limited.  One  exercise 
was  the  author's  first  simple  comparison  of  mean  Sj^eciality  Knowledge 
Test  (SKT)^  scores,  which  essentially  showed  no  diherences  between 
the  performance  of  OJT  and  TTS  graduates  on  the  SKT  for  the  Communica- 
tions Center  Operations  specialty.  This  may  have  been  indicative  of  future 


5 DiinHain,  1972 
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results  but  lacked  depth  in  that  it  did  not  account  for  related  variables 
that  may  have  some  concomitant  effect  on  SKT  scores,  such  as  the 
pre-training  abilities  of  the  trainees. 

Another  AFHRL  study,  The  Road  to  Work:  Technical  School 
Training  or  Directed  Duty  Assignment ? , by  Mr.  W“.  B.  Lecznar, 
usecf  several  criteria  which  are  thoimht  to  be  related  to  performance, 
including:  Job  Difficulty  Composite^  Number  of  Tasks  Performed 
(range  1-372)7,  Average  Task  Difficulty  per  Unit  Time  (range  1-7)7, 
and  operationally  prepared  Airman  Performance  Report  (range  1-9) 
overall  evaluations.  Several  variables  thought  to  cause  variation  in 
these  criteria  were  used  in  addition  to  OJT/TTS  as  grouping  in  an 
analysis  of  covariance  or,  "...  a two  treatment-one  concomitant 
variable  multiple  regression  model.  Analyzing  these  results, 

Mr.  Lecznar  concluded  that,  "The  inclination  is  to  say  that  in  these 
specialties  formal  resident  technical  training  provides  little  or  no 
advantage  over  on-the-job  training.  "9 

Airman  Proficiency  Ratings  (APR’s)  would  appear  to  be  a 
'natural*  lOr  measuring  some  aspects  of  performance,  but  they 
currently  suffer  from  such  a high  degree  of  inflation  that  their 
lack  of  variation  severely  limits  their  potential  for  this  purpose. 

The  Navy  has  also  done  some  interesting  work  in  developing 
productivity  indices.  This  concept  may  eventually  be  a fruitful 
avenue  for  future  work  but  a few  shortcomings  must  be  overcome. 

One  problem  is  that  it  will  be  difficult  to  account  for  changes  in  the 
quality  of  output  or  in  the  type  of  output.  In  addition,  output  level 
in  many  work  centers  in  the  Air  Force  is  strongly  related  to  the 
demand  for  the  output,  which  could  be  a troublesome  variable  to 
account  for  in  a productivity  index. 

A logical  attack  for  this  problem  would  be  to  select  the  best 
measure  of  productivity,  performance,  or  proficiency,  a priori, 
and  then  accept  the  results  obtained  with  observations  on  that 
measurement.  State-of-the-art  for  this  type  of  measurement  is 


0 Meade  & Christal,  1970 

7 Lecznar,  1972,  p.  2 

8 Ibid,  p.  3 

9 Op.  Cit. , p.  9 

10  BUNAVPERS,  1969 
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still  primitive,  so  measurements  used  are  those  available,  which 
are  usually  designed  for  another  purpose.  This  is  true  of  Specialty 
Knowledge  Tests  (SKT’s). 

SKT's  are  used  operationally  by  the  Air  Force  as  promotion 
criteria.  They  are  specialty-specific  paper  and  pencil  tests  updated 
frequently  by  Air  Force  supervisors  with  current  experience  in  the 
specialty.  An  SKT  score  for  an  individual  may  be  said  to  indicate 
that  individual’s  potential  for  performance  or  productivity  within 
the  given  specialty!  SKT's  are  taken  for  promotion  purposes,  thus 
all  with  a score  for  a particular  SKT  had  a common  motive  for 
excelling  on  the  test. 

If  either  OJT  or  TTS  were  'superior'  to  the  other,  one  would 
expect  this  to  be  manifested  in  the  graduates’  performance.  Studies 
discussed  above  found  no  evidence  to  support  the  hypothesis  that 
OJT  and  TTS  differ  in  the  quality  of  their  graduates.  Because  SKT’s 
are  specialty-specific,  used  by  the  Air  Force  for  promotion,  and  are 
taken  by  both  OJT  and  TTS  graduates  with  common  motives  for 
excelling,  SKT’s  may  be  useful  for  examining  this  hypothesis. 

Thus,  SKT  scores  were  used  to  compare  the  specialty  knowledge 
of  OJT  and  TTS  graduates  who  received  initial  upgrade  training  in  the 
same  specialties  for  which  OJT  cost  data  has  been  or  is  being  collected. 
The  ^’esearch  hypothesis  tested  was  that  OJT  and  TTS  graduates  are 
of  equal  quality.  Operational  testing  of  this  hypothesis  was  conducted  by 
comparing  the  individual  SKT  scores  of  OJT  graduates  with  those  of 
TTS  graduates,  holding  constant  other  factors  thought  to  be  related 
to  SKT  scores.  The  rationale  for  including  or  excluding  'related' 
variables  will  not  be  developed  here.  Analysis  of  covariance  was 
used  to  test  the  operational  hypothesis  and  also  to  test  whether  the 
'related'  variables  explained  some  of  the  variance  of  the  SKT  scores. 
Table  7 lists  the  variables  included  for  each  test  specialty.  Separate 
analyses  were  conducted  lor  the  specialties  listed  in  Table  8. 
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Table  7 


Variables  Used  in  the  Analysis  of  Covariance 

Variables 

Description 

SKT  Percent  Right 

Individual's  first  score  for  Test  Specialty 

TESTIM 

Days  between  date  of  enlistment  and  date 
of  test  administration 

NEGRO 

Categorical:  1 if  Negro,  0 if  other 

FEMALE 

Categorical:  1 if  Female,  0 if  other 

ADMINISTRATIVE  AI 
GENERAL  AI 
MECHANICAL  AI 

Aptitude  indices  as  computed  from  the 
Airman  Qualifying  Exams  taken  prior 
to  enlistment.  Values  are  in  raw 
score  form. 

TEST  REVISION  NUMBER 

SKT's  are  systematically  updated.  This 
is  a set  of  categorical  variables  in- 
dicating which  test  form  an  individual 
received  a score  for. 

OJT 

Categorical  variable  indicating  method  of 
initial  training. 

1 of  OJT,  0 if  TTS 

O.TT  X ADMIN  AI 

Interaction  between  OJT  variable  and 
Admin  AI  score 

TTS  X ADMIN  AI 

Interaction  between  TTS  variable  (1  if  TTS, 
0 if  other)  and  Admin  AI  score 

OJT  X GEN  AI 

Interaction  as  above 

TTS  X GEN  AI 

Interaction  as  above 

OJT  X MECH  AI 

Interaction  as  above 

TTS  X MECH  AI 

Interaction  as  above 
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Table  8 


Air  Force  Specialties  Analyzed 

Air  Force  Specialty  Code^^ 

Description 

551X0 

Pavement  Maintenance 

571X0 

Fire  Protection 

622X0 

Cook 

631X0 

Fuels,  petroleum 

647X0 

Materiel  facilities 

702X0 

Table  9 

Administrative  SjDecialist 

Null  Hypothesis 

Summary  of  Hypotheses 

Variables  Tested 

1 

OJT  X (appropriate)  AI 
TTS  x(appropriate)  AI 

2 

OJT 

3 

OJT  X (appropriate)  AI 
TTS  X (appropriate)  AI 
OJT 

11  Also,  Test  Specialty.  Irhe  'X'  in  each  code  is  replaced  by  a 4 or  5 
depending  on  whether  the  SKT  was  for  promotion  to  E4  or  E5  in  that 
specialty.  Both  of  the  4 and  the  5-level  SKT’s  were  used  for  all 
specialties  in  the  Table. 
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For  each  test  specialty  three  regressions  were  computed  in  the 
following  order  and  format; 

1.  SKT  = Bi(OJT)  + B2(NEGR0)  + B3(FEMALE)  - B4(OJTxAI)  ^ 

B5(TTSxAI)  ‘ BeCTESTIM)  * B7(TEST  FOltM) 

2.  SKT  = Bi(OJT)  ^ B2(NEGR0)  - B3(FEMALE)  • B0(TESTrM)  + 

B7(TEST  form)  - Ba(AI) 

3.  SKT  = B2(NEGR0)  - B3(FEMALE)  - BeiTESTIM'  ^ B7(TEST  FORM)  ^ 

B8(AI) 

Three  F tests  were  computed  based  on  the  increase  in  error  sums  of 
squares  between,  respectively,  regressions  1 and  2,  2 and  3,  and  1 
and  3. 

Null  hypothesis  number  1 concerned  whether  inclusion  of  the  two 
interaction  variables,  'OJT  x AI'  and  'TTS  x AT,  resulted  in  signifi- 
cant reduction  in  the  error  sum  of  squares.  These  variables  provide 
for  the  nossibility  that  ability,  as  measured  by  an  appropriate  Aptitude 
Index,  has  a differential  impact  on  SKT  scores  obtained  by  OJT^and 
TTS  graduat^T  This  ii  also  sometimes  called  a 'slope'  test. 

The  second  hypothesis  concerned  whether  the  OJT  dummy 
variable  significantly  improved  the  error  sum  ol  squares.  The  re- 
gression did  include  an  appropriate  Aptitude  Index  score,  but  not  in 
interaction  form.  This  is  the  'intercept'  test.  Rejection  of  the  null 
hypothesis  would  support  a contention  that  OJT  and  TTS  graduates 
have  different  average  SKT  scores,  holding  other  variables  constant. 

The  final  null  hypothesis  provided  an  'overall'  test.  It  tested 
whether  inclusion  of  both  the  interaction  variables  and  the  OJT  variable 
resulted  in  a significant  reduction  in  the  error  sum  of  squares. 


T2  The  technique  us^  i^Miputationally  similar  to  that  discussed  in 
Lecznar,  1972.  Wonnacott  & Wonnacott,  )ip  77-79  and  Johnston, 
pp  192-207,  contain  discussions  of  analysis  of  covariance  using 
multiple  linear  regression  as  a computational  approach. 
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Table  10 


F Ratios  for  Test  Specialties  for  Which  the  Reductions 
in  Error  Sums  of  Squares  Due  to  OJT/TTS  Variables  were  not 
Caustically  Significant! 3 


Test  AFSC 

H 

o 

^^NUM 

^^DENOM 

^RATIO 

55140 

1 

1 

121 

2.42 

2 

1 

122 

3.30 

3 

2 

121 

2.88 

55150 

1 

1 

1494 

0.07 

2 

1 

1495 

•0.65 

3 

2 

1494 

0.38 

57140 

1 

1 

2710 

0.27 

2 

1 

2711 

0.89 

3 

2 

2710 

0.57 

57150^^ 

1 

1 

1914 

0.07 

2 

1 

1915 

4.37 

3 

2 

1914 

2.22 

62240 

1 

1 

621 

2.46 

2 

1 

622 

1.32 

3 

2 

621 

1.89 

6225014 

1 

1 

2982 

4.  19 

2 

1 

2983 

5.77 

3 

2 

2982 

4.98 

63150^4 

1 

1 

1428 

1.30 

2 

1 

1429 

4.03 

3 

2 

1428 

2.67 

64750^ 

1 

1 

3909 

5.55 

2 

1 

3910 

2.38 

3 

2 

3909 

3.97 

702502 

1 

1 

6458 

0.61 

2 

1 

6459 

0.28 

3 

2 

6458 

0.44 

13  At  the  . 01  level 

— 

14  TESTIM  was  included  in  these  regressions.  This  variable  was  excluded 

from  other  F tests  because  it  was  not  statistically  significant  at  the  .01 

Test  Specialties  for  which  OJT/TTS  Variable 
Proved  to  be  Statistically  Signif icant-*-^ 


served  only  in  the  specialty  for  which  that  SKT  was 
completed  under  Project  6323,  Task  0209,  Study  4903. 


Table  9 summarizes  these  hypotheses.  Table  10  lists  the  test 
specialties  for  which  the  F ratios  would  not  allow  rejection  of  the 
null  hypotheses.  For  these  test  specialties’  no  differences  were  found 
between  the  SKT  scores  of  OJT  graduates  and  those  of  TTS  graduates. 
Table  11  displays  regression  and  correlation  statistics  for  the  test 
specialties  for  which  one  or  more  of  the  null  hypothesis  was  tech- 
nically rejected.  Figures  1,  2,  and  3 show  the  data  of  Table  11 
graphically,  holding  constant  the  dummy  variables  NEGRO,  FEMALE, 
and  TEST  FORM. 

Figure  1 illustrates  the  results  for  SKT  test  specialty  63140, 

Fuels  Specialist,  Acceptance  of  the  second  hypothesis  means  that 
there  was  a statistically  significant  difference  in  the  performance 
of  OJT  and  TTS  graduates  on  the  63140  SKT.  Ability  as  measured 
by  the  Mechanical  Ai  apparently  has  the  same  relationship  with  SKT 
scores  for  both  OJT  and  TTS  graduates  in  this  specialty. 

The  same  results  obtain  for  test  specialty  64740,  Materiel 
Facilities  Specialist,  except  the  difference  in  mean  scores  as 
evidenced  by  the  ”OJT"  coefficient  is  slightly  larger.  The  display 
would  be  similar  to  Figure  1. 

Test  specialty  70240,  Administrative  Specialist,  is  interesting 
in  comparison  because  not  only  is  the  mean  SKT  score  higher  for 
”OJT"  graduates,  but  also  ability  may  have  a differentiafTm^act  on 
OJT  and  TTS  graduates.  Figure  2 illustrates  these  results. 
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Figure  1 
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Figure  2 
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Figure  3 

all  OTHER  SPECIALTIES  ANALYZED 
^T  OJT 


1.27 


Figure  3 shows  how  the  other  test  specialties  would  have  appeared 
on  the  graphs  - no  difference  in  intercept  or  slope.  The  OJT  and  TTS 
lines  are  coincident. 

The  differences  in  SKT  scores  found  for  OJT  and  TTS  graduates 
are  statistically  significant,  but  small.  Larger  differences  might 
have  justified  a strong  statement  concerning  the  quality  of  OJT  rela- 
tive to  TTS.  SKT  scores  at  i)cst  measure  only  performance  potential 
without  considering  other  ^l•l^none^ts  of  performance,  so  small  dif- 
ferences  in  STCT  scores  i-y  uidiiselves  liave  limited  implications. 

These  results  provide  no  evidence  to  support  a contention  that  OJT 
and  TTS  graduates  differ  in  quality. 

V.  CONCLUSIONS 

Work  completed  and  underway  at  AFHRL  is  providing  the  Air  Force 
with  OJT  cost  estimates  compatible  for  comparison  with  TTS  cost  esti- 
mates. This  is  a valuable  first  step  but  these  cost  data  have  limited  use 
by  themselves.  They  must  be  used  in  conjunction  with  measures  of 
other  criteria. 

Quality  of  training  received  is  one  of  the  criteria  not  incorporated 
into  the  cost  estimates.  Individual  performances  on  Specialty  Knowledge 
Tests  were  used  in  this  paper  to  compare  the  quality  of  OJT  and  TTS 
graduates  for  a limited  number  of  specialties.  The  differences  in  SKT 
scores  found  using  analysis  of  covariance,  were  too  small  to  allow 
rejection  of  the  research  hypothesis  that  OJT  and  TTS  graduates  are 
of  equal  quality.  There  is  clearly  no  alarming  difference  in  quality 
as  measured  by  SKT  scores  for  the  specialties  analyzed.  One  may 
not  generalize  either  the  cost  results  on  the  SKT  results  to  other 
specialties. 

Future  work  concerning  OJT/TTS  tradeoffs  should  include  improved 
OJT  cost  estimation,  repetitions  of  the  SKT  analyses  of  this  paper  for 
other  specialties,  development  of  a computerized  algorithm  for  assigning 
non-prior  service  personnel  to  OJT  and  Technical  Training  Schools,  and 
development  of  metliodologies  for  determining  the  capacity  of  units  to 
conduct  OJT.  This  last  item  deserves  special  emphasis  because  it  is 
the  key  to  efficient  utilization  of  Air  Force  training  resources. 

Air  Force  operational  units  have  manpower  standards  against 
which  OJT  trainees  are  drawn.  Each  additional  trainee  replaces  a 
more  qualified  individual  imd  requires  the  time  of  other  personnel 
to  conduct  OJT.  Assignment  of  an  excessive  number  of  trainees 
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would  result  in  unacceptable  degradation  of  unit  productivity.  Even 
if  OJT  were  less  "costly”  and  produced  equally  well  "qualified" 
graduates,  there  is  a limit  beyond  which  either  TTS  would  have  to 
be  used  or  else  manning  standards  would  have  to  be  changed. 

Training  costs  form  a large  part  of  the  DoD  budget.  Since  the 
near  future  holds  only  budget  cuts,  one  may  anticipate  increased 
interest  in  establishing  procedures  and  data  bases  for  selecting 
optimal  OJT/TTS  mixes.  AFHRL  research  in  this  area  provides  a 
sound  basis  for  continuing  development  of  these  methodologies. 
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APPENDIX  A 

OJT  Cost  Survey  as  sent  to  USAF 
supervisors  with  trainees  upgrading  to 
the  3(semiskilled)- level  in  the  first  five 
specialties  listed  in  Table  8.  The  surveys 


FOR  OFFICIAL  USE  ONLY 


OJT  COST  SURVEY 

AFPT  80-5X6X-109 


This  survey  is  part  of  AF  Contract  #F41609-72-C-0048, 
for  which  Personnel  Research  Division  of  the  AF  Human 
Resources  Lab  is  the  contract  monitor. 


DEPARTMENT  OF  THE  AIR  FORCE 

AFHRL  PERSONNEL  RESEARCH  DIVISION  (AFSCi 
LACKLAND  AIR  FORCE  BASE  TEXAS  78236 


REFIT  TO 
ATTN  OF 

SUBJECT 

TO: 


PESE  (Capt  Dunham,  4106) 
OJT  Cost  Survey 


MAV  .1  1 ig?3 


OJT  Supervisors 

1.  The  purpose  of  the  attached  survey (s)  is  to  collect  data  concerning 
On-Che-Job  Training  to  the  3-skill  level.  This  survey  data,  along  with 
information  from  other  sources,  will  be  used  in  decisions  concerning  OJT 
and  Technical  Training  School. 


2.  Answering  the  survey  questions  with  some  thought  and  effort  will  aid 
Air  Force  decision  makers  in  the  management  of  your  AFSC. 

3.  Permission  to  conduct  this  survey  was  granted  by  Hq  USjiF/DP.XOS , 
reference  Air  Force  Personnel  Test  (AFPT)  Number  80-5X6X-109. 


FOR  THE  COMMANDER 


Chief,  Personnel  Research  Division 


1 Atch 
Survey 
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INSTRUCTIONS  TO  OJT  SUPERVISORS 


The  accompanying  survey  is  part  of  a research  effort  directed  toward 
evaluating  the  costs  and  benefits  of  "On  the  Job  Training."  Your  cooperation  in 
completing  the  survey  is  requested.  While  it  will  probably  take  less  than  an  hour  of 
your  time,  the  information  you  provide  will  be  very  valuable  to  the  research  and  will 
help  to  improve  Air  Force  policies  concerning  OJT  and  Technical  Training  School. 

If  you  do  not  quite  understand  a question,  give  the  best  answer  you  can  and 
feel  free  to  write  in  an  explanatory  comment  next  to  the  question  or  on  the  back  of 
the  form.  If  you  are  completely  uncertain  about  what  a question  means,  enter  a "?". 

If  a question,  for  some  reason,  does  not  apply  to  your  unit,  enter  "N.A," 

The  survey  is  divided  into  two  parts:  A and  B.  Part  A asks  you  to  try  to 
make  the  best  estimates  you  can  about  your  average  experience. 

Part  B asks  you  to  keep  a record  of  activities,  each  day  for  a week.  It  is 
important  that  you  do  this  daily,  so  that  what  was  actually  done  is  fresh  in  everyone's 
mind.  If  you  also  feel  that  the  week  you  reported  on  is  not  representative  of  your 
normal  operations,  so  indicate  by  writing  in  an  appropriate  comment;  and  if  you  can, 
indicate  what  the  average  value  ought  to  be  in  your  judgment. 

If  you  have  any  questions,  contact  Capt  Dunham,  Autovon  473-4106, 
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1 


Q Number 


Upgrade  Training  in  AFSC  [ 


SPECIAL  INSTRUCTIONS 

1 . The  trainee's  supervisor  should  complete  this  survey.  Approximately  one  half  (1/2) 
hour  will  be  required  to  complete  Part  A,  and  five  minutes  per  day  for  o week  will  be 
necessary  for  Part  B. 


2.  When  answering  the  questions,  be  sure  to  have  a Job  Proficiency  Guide  (STS),  and 
the  Consolidated  Training  Record  AF-623  for  each  person  undergoing  training,  handy  to 
refer  to. 


3.  The  person  who  fills  out  this  survey  is  encouraged  to  ask  for  the  help  of  others,  such 
as  the  OJT  Monitor  or  an  instructor  when  uncertain  about  the  answer  to  a question. 


4.  Part  A which  should  be  completed  immediately,  is  to  be  returned  together  with  Part 
B within  8 days.  Do  not  start  Part  B before  completing  Part  A. 

5.  If  there  is  difficulty  in  deciding  what  information  is  being  asked  for  in  any  question, 
contact  Capt  Dunham,  Autovon  473-4106. 
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GRADE  If  Air  Force  NCO  enter  "4"  for  E4,  "5"  for  E5,  etc. 
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PART  A 


1 . How  many  trainees  do  you  have  upgrading  to  the  3 and  5 level  in  your 
section? 

2,  When  a man  (or  woman)  first  reports  directly  from  Basic  Military 
Training,  it  may  take  some  time  before  he  actually  begins  training  and 
work,  even  though  his  "date  of  entry"  to  training  may  be  the  same  as  his 
reporting  date.  This  delay  may  be  due  to  personnel  processing,  the  need  to 
wait  for  security  clearance,  or  some  other  cause.  Approximately  how  many 
days  does  it  take  before  the  newly  arrived  "helper"  actually  begins  OJT? 


□ 

3 level  5 level 
trainees  trainees 


work  days 


3.  There  is  also  delay  in  entering  training  associated  with  the  arrival  of 
a 3 level  from  Technical  School.  In  addition  to  personnel  processing,  ___ 

familiarization  with  procedures  specific  to  your  situation  may  be  necessary  | 

before  he/she  actually  begins  5 level  training.  On  the  average,  this  delay  work  days 

is: 


4.  On  the  average,  how  many  weeks  elapse  between  achievement  of  3 
level  proficiency  and  actual  award  of  the  3 skill  level  AFSC? 

5.  What  week  of  training  is  your  most  average  1 level  (helper)  in? 

6.  What  % of  the  3 level  proficiency  training  do  you  estimate  he  has 
completed? 

7.  When  he  arrived  what  % of  the  duties  of  a 3 level  could  he  complete? 

8.  If  you  stopped  doing  OJT  training  would  you  be  able  to  reduce  the 
number  of  NCO's  in  your  work  area  without  significantly  reducing 
effectiveness?  (Insert  a "1"  for  Yes,  or  a "0"  for  No). 

9.  During  the  training  period  for  3 level  OJT,  the  instructor  (trainer) 
must  spend  some  time  keeping  training  records  up  to  date.  On  the 
average  over  the  whole  training  period,  how  many  hours  (or  fractions  of 
hours)  per  week  does  the  instructor  (trainer)  spend  in  record  keeping  for 
one  trainee? 


Q 


we^s 


[ I i 

week 


Hrs . 
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Part  A (cont'd) 

10.  The  newly  arrived  Tech  School“trained  3 level  is  not  as  productive 
at  first  as  the  OJT-trained  3 level  is,  although  he  may  soon  close  the  gap. 

a.  In  your  estimate,  what  percentage  of  the  workload  of  on  OJT“ 
trained  3 level  can  the  Tech  School  graduate  handle  Immediately 
after  his  arrival? 

b.  How  many  weeks  does  it  take  before  the  Tech  School-trained 
3 level  works  with  as  little  supervision  as  an  OJT“trained  3 level? 

c.  After  both  types  of  3 levels  are  awarded  their  5 level,  on  the 
average  do  you  consider  either  to  have  superior  performance? 

(Insert  a "1"  for  Yes,  or  a "0"  for  No.) 

d.  If  your  answer  was  " which  type  of  3 level  do  you 
consider  to  have  better  performance?  (Insert  a "1"  for  OJT,  or  a 
"0"  for  Tech  School.) 

11.  If  extra  (remedial)  training  is  conducted  in  your  office  for  trainees 
who  fail  the  End  of  Course  Exam  (Apprentice  Knowledge  Test),  answer 
the  following  questions: 

a.  In  your  experience,  what  percent  of  the  3 level  trainees  fail 
the  End  of  Course  Test  the  first  time  they  take  it? 

b.  On  the  average,  how  mat'iy  weeks  of  additional  training  are 
given  to  airmen  who  fail  the  End  of  Course  Exam  before  they  take 
the  test  again? 

c.  How  many  hours  per  week,  during  the  normal  work  week,  does 
the  trainee  spend  engaged  in  this  remedial  training? 

d.  How  many  hours  per  week,  luring  the  normal  work  week,  does 
the  instructor  spend  engaged  in  this  remedial  training? 

12.  If  you  stopped  doing  OJT  training  and  had  no  replacements  for  the 
trainees  could  your  section  continue  to  perform  its  mission  without 
significantly  reducing  effectiveness?  (Insert  a "1  for  Yes,  or  a 0 for 
No.) 
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Part  A (cont'd) 


13.  Based  on  your  past  experience,  and,  if  you  feel  you  need  help,  the  experience  of 
other  qualified  personnel  in  your  section,  list  the  average  number  of  productive  and 
non-productive  hours  of  work  for  the  trainee  upgrading  to  the  3 level  for  each  week 
between  start  of  training  and  award  of  skill  level . For  instance,  in  the  fourth  week  of 
training  your  trainee  spent  approximately  30  hours  receiving  instruction  and  reading  and 
10  hours  doing  productive  work.  Your  second  entry  would  look  like  this: 


4 


Note  that  the  hours  for  each  week  must  sum  to  40,  and  you  must  have  an  entry  in  every 
week.  If,  on  the  average,  trainees  complete  training  between  the  12th  and  16th  week, 
then  the  entry  for  the  sixteenth  week  should  show  a "40"  under  productive  and  a "0" 
under  instruction. 


Weeks  of  Training 
(to  the  3-level) 


Trainee  Productive 
Hrs  Per  Week 


Instruction  & Reading 
Hrs  Per  Week 


1 n ! 

L..  .,  ■ j 
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1 

16 
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20 
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14.  What  is  the  total  number  of  personnel  in  your  section  (officer,  enlisted, 
and  civilian)? 

15.  In  addition  to  the  trainees  you  now  have  responsibility  for,  how  many 
more  1 level  trainees  could  your  section  train  right  now  without 
significantly  reducing  the  effectiveness  of  section  operations? 

(ignoring  the  limit  on  authorized  number  of  personnel) 


no.  of  pers. 


n 

1 level 
trainees 


16.  If  you  had  to  lose  a qualified  5 level  for  each  new  1 level  trainee 
(helper),  how  many  more  1 level  trainees  could  your  unit  train  right  now 
without  significantly  reducing  the  effectiveness  of  section  operations? 
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PART  B 


FOR  ONE  WEEK  PLEASE  KEEP  A RECORD  AT  THE  END  OF  EACH  DAY  OF  THE 
AMOUNT  OF  TIME  SPENT  IN  EACH  CATEGORY. 


17.  How  many  DDA  airmen  do  you  currently  have  enrolled  in 
OJT  to  the  3 level? 


Airmen 


18.  Record  daily  the  total  hours  your  1 level 
trainees  spend  on  reading  and  receiving 
instruction  each  day.  (Be  sure  and  ask  your 
trainees  for  their  assistance  in  completing 
this  question). 

19.  Record  daily  the  total  hours  your  1 level 
trainees  spend  in  activities  contributing  to 
office  productivity. 

20.  Record  daily  the  total  hours  of 
instruction  provided  by  each  grade  of 
instructor. 


Mon  T ues 

m i:x 


AFTER  YOU  HAVE  COMPLETED  THE 
ENTRIES  FOR  FIVE  DAYS  RETURN 
THE  SURVEY  TO  YOUR  BASE 
CBPO 
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SYMPOSIUM 


TRANSLATION  OF  TRAINING  RESEARCH  INTO  TRAINING  ACTION  - A MISSING  LINK 


Introductory  Remarks 
G.  Douglas  Mayo 

Naval  Technical  Training  Command 


I accepted  the  chairman's  job  in  this  symposium  with  the  understanding 
that  I would  not  have  to  be  the  kind  of  chairman  who  is  neutral  or  im- 
partial, but  instead  that  I could  expose  my  biases  on  the  Translation 
of  Training  Research  into  Training  Action,  the  same  as  anyone  else.  So, 
in  introducing  the  topic  it  should  not  be  surprising  if  I express  some 
of  the  thoughts  that  I may  have  been  repressing  during  the  two  decades 
that  I have  been  Involved  in  training  research  and  in  translating  it 
into  training  action. 

During  this  period,  we  have  won  a few.  We  are  all  familiar  with  in- 
stances in  which  the  results  of  training  research  and  development  have 
been  implemented  into  the  on-going  training  operation  and  have  contrib- 
uted materially  to  it.  Two  examples  that  readily  come  to  mind  in  Naval 
Technical  Training  (the  area  of  operation  with  which  I am  most  familiar) 
are  programmed  instruction  during  the  decade  of  the  60 's  and  the  imple- 
mentation of  computer  based  instruction  which  is  underway  on  a substan- 
tial scale  at  the  present  time. 

But  for  every  R&D  project  that  has  had  an  impact  upon  training  there  are 
numerous  ones  that  have  not.  Now  it  can  be  argued  that  this  is  the  nature 
of  the  beast,  that  risk  taking  is  an  inherent  and  necessary  part  of  the 
R&D  process.  No  doubt  this  is  true,  but  I submit  that  the  normal  risks 
associated  with  R&D  do  not  even  begin  to  account  for  the  unused,  and  in 
some  instances  unusable,  volume  of  training  research. 

Much  of  the  systematic  civilian  work  in  the  area  of  translating  educa- 
tional research  into  educational  change  has  been  accomplished  under  the 
rubric  of  "linear  change  models  in  education."  Most  of  these  models 
describe  a linear  sequence  of  functions  that  include:  research  (the  func- 
tion in  which  new  knowledge  is  produred),  development  (in  which  a product 
or  procedure  based  on  the  new  knowledge  is  engineered  and  evaluated) , dif- 
fusion (in  which  the  generality  and  extent  of  applicability  of  the  product 
or  procedure  is  explored),  and  adoption  (the  function  of  implementing  an 
appropriate  form  of  the  product  or  procedure  in  an  on-going  educational 
situation) . It  is  generally  conceded  that  in  order  for  such  a model  to 
work,  at  Least  one  of  two  conditions  must  exist;  either  the  people  in- 
volved in  the  linear  sequence  of  functions  (i.e.  research,  development. 
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diffusion,  and  adoption)  must  be  intrinsically  motivated  toward  the 
common  goal  of  improving  education  by  means  of  research,  or  they  must 
be  extrinsical] V motivated  by  accountability  with  respect  to  accom- 
plishing the  common  goal.  More  often  than  not,  a combination  of  both 
intrinsic  and  extrinsic  factors  are  at  work  in  varying  degrees  in  the 
instances  in  which  the  linear  change  model  functions  properly. 

Moving  to  a more  specific  level,  and  with  special  emphasis  upon  the 
military  situation,  there  are  a number  of  conditions  that  contribute 
to  the  "missing  link"  in  translating  training  research  into  training 
action.  For  convenience  they  may  be  divided  into  three  groups,  namely, 
those  associated  with  research  personnel,  those  associated  with  train- 
ing personnel,  and  the  Interface  between  research  personnel  and  train- 
ing personnel. 

First,  research  personnel.  Not  infrequently  research  personnel  conceive 
of  a "good"  study  and  then  look  for  a training  situation  in  which  to  do 
it,  without  seriously  considering  whether  the  findings  of  the  study,  if 
successful,  could  be  Implemented  in  that  or  any  other  school.  In  my  ex- 
perience, it  is  not  as  unusual  as  one  would  wish  for  a researcher  to 
complete  a training  research  project  in  which  his  hypotheses  are  sustained, 
and  yet  be  unable  to  state  in  what  way  his  finding  can  be  used  to  Improve 
training  or  training  efficiency. 

A simple  solution  to  the  problem  would  seem  to  be  to  state  in  advance  for 
each  applied  research  project  what  action  should  be  taken  in  the  event 
that  any  one  of  several  possible  outcomes  of  the  study  should  result. 
Carried  to  its  logical  conclusion  this  should  prevent  the  researcher's 
wondering  what  to  do  with  his  results,  upon  completion  of  his  research. 
Sometimes  the  researcher  does  not  wonder  what  to  do  with  his  results 
because  he  does  not  feel  that  it  is  his  responsibility  to  point  out 
practical  implications  of  his  research,  that  his  responsibility  is  lim- 
ited to  properly  conceiving,  designing,  conducting,  and  reporting  the 
study . 

What  about  training  personnel  then?  Are  they  not  eagerly  awaiting  any 
research  results  that  will  improve  their  training  operation?  Well,  yes 
and  no,  but  mostly  no.  They  are  busy  with  the  day  to  day  pressures  of 
conducting  training.  Both  having  the  research  conducted  in  their  school 
and  implementing  the  results  tend  to  disrupt  the  training  operation  and 
pile  additional  work  on  a staff  that  already  feels  that  it  is  overloaded. 
Often  training  personnel  feel  that  the  R&D  was  not  their  project,  that 
any  credit  tliat  might  accrue  from  it  is  likely  to  go  to  the  researchers 
or  to  a higher  echelon  in  the  training  organization,  and  it  is  just  hard 
for  them  to  get  very  enthusiastic  about  it.  Besides,  several  months  of- 
ten elapse  between  the  time  the  data  were  collected  and  the  time  the  more 
or  less  unintelligible  report  arrives. 
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This  leads  us  Lo  our  third  set  of  conditions  that  tends  to  affect  the 
production  of  usable  research  and  the  proper  use  of  it,  the  interface 
between  research  personnel  and  training  personnel . This  may  well  be 
the  area  in  which  the  greatest  source  of  our  trouble  lies.  Too  often 
research  personnel  do  not  accompany  the  research  report  when  it  is 
submitted  to  training  people,  do  not  make  a presentation  to  training 
people  concerning  it,  and  do  not  assi.,t  in  its  implementation. 

Clearly,  it  takes  much  more  initiative  on  the  part  of  the  training 
people  to  decide  to  make  changes  in  their  training  operation  on  the 
basis  of  a report  than  it  does  to  reach  a decision  to  take  action 
when  tlie  author  of  the  report  describes  the  study,  clarifies  points 
that  are  not  understood,  and  makes  logical  recommendations  in  a face 
to  face  situation  to  the  people  who  are  responsible  for  efficient 
training  in  the  course  concerned.  The  probability  of  successful 
implementation  is  further  enhanced  if  the  researcher  continues  to 
work  with  cognizant  training  personnel  throughout  the  implementation 
of  the  research  results.  Normally  this  does  nut  require  a great  deal 
of  time  and  in  my  view,  at  least,  is  time  better  spent  than  in  uti- 
lizing this  same  time  on  another  research  study,  with  tlie  probability 
that  the  results  of  neither  of  the  two  studies  Vvri  1 1 be  implemented. 

We  have  long  known  that  in  implementing  any  concept,  responsibility 
for  its  occurring  must  be  fixed.  I am  therefore  suggesting  that  each 
organization  engaged  in  applied  R&D  should  have  a small  group,  report- 
ing directly  to  the  Technical  Director  or  to  the  Commanding  Officer, 
whose  primary  responsibility  is  two-fold,  first,  to  look  externally 
to  impl ementat ioii  of  RiD  products  and,  second,  to  look  internally  to 
the  design  and  production  of  products  that  are  capable  of  being  im- 
plemented. Thus,  on  the  one  hand  this  group  would  ensure  that  R&D 
products  were  properly  presented  to  appropriate  training  personnel, 
normally  by  the  research  personnel  who  did  the  work,  and  that  assis- 
tance in  Implementing  the  products  was  continued  as  long  as  appropriate. 
On  the  otlier  hand,  prior  to  approval  of  an  applied  research  project  the 
group  would  assess  the  probable  saleability  of  the  pnniuct  that  the  R&D 
project  was  likely  to  produce  and  so  advise  the  Teihnical  Director  or 
Commanding  Officer.  If  the  purpose  of  tlie  priijeci  was  to  produce  a 
usable  product  and  the  proposed  project  was  jiulged  not  to  be  capable 
of  proilucing  such  a product,  it  would  be  referred  back  to  the  origi- 
nator for  revision. 

At  this  point  1 will  give  the  other  members  cjf  the  panel  a chance  to 
express  their  thoughts,  repressed  or  otherwise.  1 hope  I hat  they  are 
going  to  tell  us  that  I am  the  only  one  who  has  observe<l  the  problems 
1 have  related,  that  everything  runs  smoothly  in  the  training  world 
with  which  they  are  familiar.  But  we  will  have  to  wait  and  see  if 
that  is  in  fact  the  case.  The  plan  for  the  symposium  is  to  have  two 
prepared  papers,  one  by  a trainer  (Mr.  Walter  McDowelll  and  one  by  a 
researcher  (Dr.  Norman  Kerr).  The  papers  will  then  he  discussed  by 
Lieutenant  Colonel  Donald  Mead  and  Dr.  William  Moonan. 
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In  addition,  if  I have  said  anything  in  my  introductory  remarks  that 
is  worth  disagreeing  with  or  otherwise  commenting  on,  I hope  that 
Colonel  Mead  and  Dr.  Moonan  will  include  this  in  their  discussion  also. 
Following  the  papers  by  the  two  discussants,  we  will  afford  an  oppor- 
tunity to  the  four  members  of  our  Reaction  Panel  to  make  a few  impromp- 
tu remarks.  Upon  completion  of  this  structured  portion,  we  will  be 
pleased  to  open  up  the  symposium  for  comments  or  questions  from  the 
floor.  Our  first  speaker  is  Mr.  Walter  E.  McDowell  from  the  Army  Train- 
ing and  Doctrine  Command  at  Fort  Monroe.  He  will  be  speaking  on  the 
basic  topic  of  Translation  of  Training  Research  into  Training  Action, 
from  the  Viewpoint  of  a Training  Manager. 


smposiuM 


"TRANSLATION  OF  TRAINING  RESEARCH  INTO  TRAINING  ACTION  - A MISSING  LINK?" 


From  the  Viewpoint  of  a Training  Manager 


Mr.  Walter  E.  McDowell 
Supervisory  Educational  Specialist 
Headquarters  US  Army  Training  and  Doctrine  Command 
Fort  Monroe,  Virginia 
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VVhen  given  the  requirement  to  propose  Human  Factors -Behavioral  Science 
research  for  the  coming  year,  we  are  faced  with  a soul-searching  situation 
in  which  we  must  ask  ourselves  two  questions;  one,  are  we  really  getting 
the  taxpayers'  dollar  benefit  from  the  work  that  has  transpired  in  the  past; 
and,  two,  will  the  research  we  are  asking  for  be  likely  to  yield  productive 
results  in  the  future.  Another  question  which  also  must  be  scrutinized  when 
it  surfaces  is,  if  we  aren't  getting  a maximum  return  on  our  investment  in 
research,  how  can  we,  as  trainers  and  managers,  translate  the  results  of 
future  research  into  better  returns  on  our  investment  dollars?  First  of  all, 
it  is  not  purely  a translation  problem.  The  process  of  bridging  the  gap 
between  the  conduct  of  research  and  the  implementation  of  research , by  the 
Army  trainer,  is  a continuing  one.  It  begins  while  the  problem  is  being 
defined  to  both  the  researchers  and  the  trainers;  it  continues  during  the 
conduct  of  the  research;  it  continues  while  the  final  report  is  being 
drafted,  and  it  still  continues  long  after  the  final  report  has  been  published. 

Let's  begin  at  the  beginning.  In  a very  real  sense,  utilization  begins  at 
the  time  the  problem  to  be  researched  is  presented.  The  training  problem  must 
be  carefully  defined  and  documented  to  insure  that  both  the  researcher  and  the 
trainer  are  speaking  the  same  language  and  tliat  the  researcher  fully  understands 
the  problem  that  the  trainer  believes  he  has.  After  the  trainer  submits  his 
request  for  research,  the  researcher  must  then  submit  a statement  to  the 
trainer  on  what  he  views  the  problem  to  be  and  how  he  intends  to  seek  out  the 
solution.  A common  understanding  and  agreement  of  the  problem  to  be  researched 
is  essential  if  a usable  end-product  is  to  be  the  result.  The  research  approach 
may  have  to  be  modified  in  the  light  of  continuing  support  requirements.  Some- 
times it  may  have  to  be  recast  if  the  research  proposed  is  not  understood  by 
the  trainer  to  be  appropriately  responsive  to  the  problem  as  he  perceives  it. 

Constant  interaction  between  researcher  and  trainer  takes  place  during  the 
conduct  of  the  work.  Communication  is  achieved  informally  on  a continuing 
basis  through  the  use  of  interim  progress  reports,  periodic  reviews,  briefings, 
and  discussions.  The  trainer  requesting  the  research  must  be  kept  constantly 
aware  of  the  work  taking  place,  and  the  researcher  must  be  kept  abreast  of 
significant  military  changes  within  the  parameters  of  training  problems.  Hope- 
fully, the  interaction  and  open  communication  will  keep  the  evolving  solution 
directly  aligned  with  the  trainer's  problem. 

As  relevant  data  becomes  available,  that  is,  data  which  bears  upon  the 
problem,  it  is  provided  to  the  trainer.  Such  communication  involves  both 
those  management  functions  which  can  implement  change  and  those  responsible 
for  actr.ally  carrying  out  the  decision  to  make  the  change.  Management,  simply 
decreeing  that  change  be  made,  will  not,  alone,  get  the  job  done;  the  "not- 
invtnted  here"  syndrome  must  be  overcome.  The  "doer"--thr  guy  at  the  working 
level--must  be  convinced  that  the  change  is  good  and  that  it  will  be  of  benefit, 
not  only  to  himself,  but  to  the  prospective  students  and  to  the  system,  other- 
wise it  will  be  doomed  to  failure. 
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When  the  report  is  submitted,  it  is  reviewed  and  sLaiied  witli  apwru- 
priate  elements  of  the  organization.  Each  staff  element  examines  and  makes 
recommendations  as  to  the  suitability  of  the  report  as  a research  document. 
Some  results  which  come  from  research  are  not  usable  in  the  formal  report 
form  because  they  may  have  been  designed  for  background  or  for  reference  use, 
or  perhaps  to  provide  a basis  for  a review  of  training  policies  or  programs. 

In  all  too  many  cases,  the  findings  must  first  be  subjected  to  a special 
translation  and  adaptation  in  order  to  become  operationally  useful.  This 
translation  is  accomplished  best  by  a joint  effort  between  researcher  and 
trainer  and  must  be  done  with  extreme  care  to  insure  that  the  research  results 
are  not  lost,  obscured,  misused,  or  inoperable  and  at  the  same  time  are 
understandable  and  usable  at  the  lowest  level  of  organization-the  units  in  the 
field-and  by  individuals  who  maybe  limited  in  their  intellectual  ability. 

Frequently  it  is  essential  to  involve  the  research  team  in  active 
participation  in  this  translation  effort.  Team  members  provide  assistance 
in  the  preparation  of  action  directives,  pamphlets,  regulations  and  in 
adapting  the  findings  to  specfic  operational  contexts.  Such  is  the  case, 
at  the  present  time,  with  the  Fuctional  Literacy  Training  (FLIT)  effort  at 
Fort  Ord , California,  and  the  ATC  PERFORM  project,  a performance-oriented 
training  program,  being  adapted  at  several  training  centers  and  the  WAC  School 
at  Fort  McClellan,  Alabama. 

Research  by-products,  such  as  experimental  course  units  or  programs  of 
instruction,  work  books,  prototype  training  aids  and  devices,  programed  text 
booklets  and  supplemental  training  literature,  are  similarly  reviewed.  It  is 
considered  that  the  incidence  of  these  by-products  being  Incorporated  into 
on-going  Army  training  efforts  occurs  with  great  and  increasing  frequency. 

Utilization  of  research  many  times  occurs  while  the  research  effort  is 
in  progress  and  prior  to  completion  of  the  project.  When  the  school  or 
training  center  that  has  research  sponsorship  is  impressed  with,  and  convinced 
of  the  value  of  the  work,  on-the-spot  implementation  may  occur  through 
incorporation  of  those  complete  segments  of  the  research  results  into  the 
on-going  training  programs. 

Results  of  research  completed  some  v^ars  in  the  past  are  often  found  to 
be  relevant  to  today's  problems.  How  to  tap  the  vast  storehouse  of  past 
research  to  preclude  re-inventing  the  wceel,  is  an  area  that  deserves  our 
immediate  attention.  Computer  and  other  retrieval  technology  has  been  with 
us  for  some  time.  We  need  an  efficient  means  of  identifying  past  efforts 
as  they  may  apply  to  todays  problems. 

Essential  to  utilization  is  the  dissemination  of  research  findings  among 
potential  users.  The  Training  and  Doctrine  Command  at  Fort  Monroe,  Virginia, 
employs  several  means  to  alert  elements  of  the  command  to  research  work,  both 
in-progress  and  already  completed.  Bulletins  to  headquarters  elements,  staff 
briefings  by  teams  representing  the  Human  Research  Units,  the  researchers 
themselves,  the  schools  and  training  centers,  and  newsletters  to  headquarters 
staffs,  centers  and  schools--all  serve  to  publicize  the  results  of  research. 
Written  quarterly  progies.s  reports  are  submitted  and  final  research  reports 
are  appropriately  distributed,  including  their  entry  into  the  Defense  Documen- 
tation Center  (DDC)  and  the  Department  of  Commerce  Clearinghouse  for  Federal 
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Scientific  and  Technical  Information  (CFSTI)  . All  important  are  the  informal 
and  continuing  contacts  between  researchers  and  potential  users. 

A report  by  Dr.  William  A.  McClelland,  at  the  Conference  on  Social 
Research  and  Military  Management  at  the  University  of  Chicago,  June  1967, 
gives  an  insight  into  possible  characteristics  of  unsuccessful  and  successful 
research  implementation;  UNSUCCESSFUL  RESEARCH  EFFORTS: 

(1)  Poor  communication.  Neither  briefings  nor  reports  effectively 
communicated  the  validity  and  operational  value  of  the  research. 

(2)  Lack  of  timeliness.  The  product  of  the  research  effort  did  not  meet 
a valid,  contemporary  requirement.  It  was  available  too  late  or  too  early, 
or  it  was  too  tangential  in  nature. 

(3)  Degree  of  change.  Too  many  changes  in  operating  procedures  were 
required.  For  example,  training  was  shortened  (or  lengthened)  too  much,  or 
the  existing  Army  structure  was  incompatible  with  the  indicated  change.  Exist 
ing  or  traditional  practice  may  have  been  too  strongly  threatened. 

(4)  Lack  of  strong  command  support.  Not  enough  people  at  high  enough 
echelons  wanted  to  cliange. 

(5)  Costs.  Funds  and  personnel  required  had  not  been  programed  and  could 
not  be  obtained. 

(6)  Lack  of  engineering  capability.  The  training  experts  required  to 
translate  the  research  findings  into  more  operationally  usable  form  did  not 
exist  or  were  not  available. 

(7)  Policy  problem.  There  was  a lack  of  doctrine  under  wtiich  to  fit  a 
new  or  improved  training  or  operational  capability. 

(d)  Insufficient  "salesmanship".  Project  people  did  not  devote  enough 
effort  to  "selling"  the  product.  At  one  time,  for  example,  we  believed  this 
was  not  the  job  of  the  research  agency. 

Possible  reasons  for  successful  implementation  are  largely  the  obverse  of 
this  list.  SUCCESSFUL  RESEARCH  EFFORTS: 

(1)  Timeliness.  A recognized  instructional  gap  was  filled.  The  work  was 
obviously  relevant  to  a planned  or  on-going  revision  in  Army  practice. 

(2)  Comnand  interest.  There  was  a strong  operational  command  interest, 
including  that  of  a subordinate  command.  Interest  existed  at  both  management 
and  working  levels. 

(3)  Engineered  product.  The  end-product  was  concrete.  It  was  a material 
plug-in  item,  specifically  engineered  for  a given  situation  requiring  little 
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additional  Army  effort  to  adapt  it  to  the  operational  setting  and  requiring 
no  doctrinal  changes. 

(4)  Earlier  acceptance  by  others.  Some  other  service  or  civilian 
institution  had  accepted  and  successfully  used  the  product  or  a very  similar 
one . 

(5)  Personal  interest.  An  individual  Army  officer  or  group  of  officers 
or  key  civilians  associated  with  the  work  were  convinced  of  its  worth  aiit!  were 
willing  to  serve  as  forceful  proponents. 
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SYMPOSIU'I 


"TRANSLATION  OF  TRAINING  RESEARCH  INTO  TRAINING  ACTION  - A MISSING  LINK?" 


From  the  Viewpoint  of  a Training  Researcher 


Dr,  Norman  J.  Kerr 
Director  of  Research 
Naval  Technical  Training  Command 
Naval  Air  Station  Memphis 
Millington,  Tennessee 
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I can  never  help  but  note  Chat,  each  time  I analyze  the  area  ot  training 
repearcli  with  respect  to  training  management,  ttie  former  comes  out  at  the  lov;er 
end  of  the  continuum  of  power.  There  are  reasons  for  this  and  not  necessarilv 
objective  ones.  And  I tend  to  lieap  a prodigious  quantity  of  the  blame  for  this 
aquarelv  upon  the  shoulders  ot  the  training  management  communitv.  Too  often, 
well-conducted  programs  of  researcli  Itave  tiie  re.su]  t that  the  researciiet  Lircomes 
impaled  upon  the  sliarpened  fence  posts  of  ineptitude,  inaction,  and  sheer 
ignorance.  It  also  seems,  at  times,  tliat  well-intentioned  research  easilv 
falls  victim  to  a well-organized  "dirty  tricks"  group  of  a well-endowed  power 
organ i zation . 

Those  of  us  in  training  research,  while  not  necessarilv  comf or C .ab] e with 
this  situation  have,  none-the-1 css , grown  accustomed  to  such  conditions  anti 
are  quick  to  sense  tiisparitles  at  all  levels  of  interaction  in  the  courses  of 
the  relationship  between  training  management  and  training  research. 

It  is  preciselv  this  disparitv,  it  seems  to  me,  that  is  the  'entral 
focus  for  the  cons  i derat  ion  of  this  svmposium.  Whv  is  it  that  training, 
researcii  and  training  researchers — and  there  is  an  oven^7helming  amount  ot 
evidence  that  this  also  miglit  be  said  for  the  behavioral  science  field  in 
toto — are  not  provided  with  tlie  proper  degree  of  status?  Tn  a societ/  in 
which  one  is  so  willing  to  accept  certain  "truths",  however  unreasonable  t he" 
may  be,  tliat  flow  from  others  who  should  know  (National  politicians  are  a 
gijoci  example),  whv  is  it  that  there  is  anv  dlfficultv  to  the  acceptance  of 
not  onlv  tile  word  but  the  irrefutable  data  of  the  training  researcher? 

Perhaps  those  of  vou  in  training  management  think  I do  you  some  injustice- 
' cannot  agree.  The  training  manager,  in  my  perception  of  him,  alwavs  seeks 
answers  yesterday  to  tomorrow's  problems  which  he  is  causing  todav.  He  tra- 
ditionally accepts  a truism  such  as  "individualization  is  'good'"  or  "train- 
ing time  must  be  reduced  bv  10%"  and,  with  unquestioning  blindness,  blunders 
forward  with  a series  of  compliance  actions.  Unquestionablv,  he  must  instinc- 
tively know  that  any  adjustments  to  the  system  will  result  in  some  varving 
degree  of  turmoil.  But  it  must  be  tliat  he  hopes  that  the  turmoil  will  be 
temporary  rather  than  permanent — and  he  does  not  willinglv  accept  the  fact 
that  such  actions  may  he  not  only  intemperate  or  maltimed  but  are  nothing 
short  of  downright  wrong.  It  might  be  that  he  is  Insecure  in  his  role  and  is 
often  driven  by  power  beyond  his  capabilities  to  control  (much  less  to  compre- 
hend), be  it  national  policies  for  the  recruitment  of  personnel  or  service 
policies  which  dictate  economic  controls  of  severe  stringency.  He  accepts  the 
truism  or  the  dictum  and  hastens  to  react.  Perhaps  he  is  intimidated  bv  the 
world  of  reality  as  he  perceives  it  and  he  feels  that  he  has  to  remain  flexible 
enough  to  get  his  job  done  in  the  face  of  such  factors.  He  Is,  in  actuality, 
a great  compromiser;  he  plugs  a hole  here,  redirects  a pipeline  flow  there, 
introduces  an  Innovation  now  and  then,  and  follows  the  rules — those  almighty 
rules — of  action,  and  keeps  his  areas  of  Interest  botli  proper  and  orderly. 


While  the  lirst  one  to  sav  that  job  analysis  must  be 
tlie  construction  of  an  effective  curriculum,  the  training 
known  to  be  capable  of  putting  together  a curriculum  on  a 
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weapons  system  (for  example)  over  night. 

The  typical  training  manager  will  fight  the  instructor  billet  battle 
seemingly  to  the  wire  bv  a continual  walling  over  inadequate  numbers  of 
instructors;  yet  he  is  verv  willing  to  accept  new  curriculum  administration 
responsibilities  for  his  overworked  school  staffs.  Those  who  question  this 
should  ask  themselves  how  many  or  in  what  proportions  were  new  instructor 
billets  provided  when  additional  curriculum  burdens  such  as  human  relations, 
defensive  driving,  and  remedial  reading  programs  were  directed  for  inclusion 
into  existing  training  programs.  Yet,  these  good  men  seem  obsessed  with 
management  and  its  precise  controls,  and  often  decline  well-intentioned 
advice  and  assistance.  That  is  putting  it  mildly,  for  it  is  onlv  with  re- 
luctance and  suspicion  that  many  training  managers  will  accept  a rationale 
or  the  experience  of  others  who  are  not  onlv  directly  in  but  senior  within 
his  line  organizational  structure. 

It  might  be  that  the  training  manager  has  the  utmost  confidence  in  him- 
self— not  necessarilv  in  his  abilities  to  improve  the  training,  but  most 
assuredly  in  his  moxie  to  be  able  to  handle  the  many  and  varied  responses 
he  will  derive  from  both  judgmentally  sound  and  highlv  questionable  decisions 
which  he  has  made  in  the  name  of  training  management.  After  all,  is  this  not 
what  training  management  is  all  about? 

Have  I left  out  planning — or  the  lack  thereof — by  the  training  manager? 

I have  not  meant  to.  As  any  manager  knows,  planning  is  an  essential  part  of 
his  daily  routine.  My  Lord,  does  he  plan  I He  has  training  plans,  personnel 
pipeline  plans,  SER  plans,  CIVSUB  plans.  . . Heaven  knov;s  how  manv  other 
schemata  he  considers  in  the  laying  out  of  training  requirements  and  in  the 
execution  of  training  activities.  But  ask  the  training  management  community 
for  research  plans  and  stand  bv  for  a high  fog  count  in  the  responses.  Better 
yet,  ask  for  definitions  of  training  research  and  you're  likely  to  get  responses 
in  the  order  of  "something  not  needed",  "something  received  after  the  fact",  and 
"something  that  is  needed  to  verify  mv  actions."  The  training  manager,  often 
willing  to  spend  hours  grumbling,  perhaps,  but  unquestionablv  counting  the  number 
of  sessions  in  a score  of  curricula  to  develop  data  to  satisfy  an  obsolete 
report  on  a hands-on  vs  theory  ratio,  is  generally  not  prone  to  review  the 
research  conducted  on  highly  pertinent  areas  of  his  responsibility.  Yet  that 
research  which  might  well  enable  him  to  better  conduct  the  business  of  training 
may  be  in  evidence  all  about  him,  gathering  the  patina  of  GSA  dust. 

Early  in  preparation  for  this  svmnosium  1 decided  that  my  assigned  topic 
"From  the  Viewpoint  of  a Training  Researcher"  gave  me  wide  latitude  in  pin- 
pointing— at  least  to  the  satisfaction  of  my  own  perceptions — where  the  problem 
exists  in  the  translation  of  training  research  into  training  actions.  I think 
that  I have  now  made  that  point  several  times  over  and  I supposed  that,  since  I 
probably  have  a goodly  percentage  of  my  listeners  in  a state  of  spirited  agitation, 
I should  smile  broadly  and  back  slowly  out  of  the  nearest  exit.  I promise  that 
I will  do  just  that  very  soon,  but  first  let  me  say  that  I tried  to  locate  strong 
support  for  my  contention  from  the  literature. 


To  my  dismay  I found  tliat  manv  of  the  articles  on  the  subject  in  research 
and  development  professional  journals  are  reallv  self-deprecating.  We  in 
research  are  really  rather  prone  to  take  on  the  problems  of  the  world;  to 
blame  ourselves  for  anotlier's  inability  or  unwillingness  to  comprehend.  The 
chest  beating  is  often,  in  realitv,  a hushed  humble  breast-thumping  whicli  marks 
the  contrite,  and  there  are  manv  in  research  who  continually  try  to  find  the 
missing  link  of  translating  training  researcli  into  training  action  alone.  it 
was  amazing  how  sparse  was  the  material  in  support  of  the  ratiier  parochial 
position  1 have  presented.  As  a matter  of  fact,  I found  the  strongest  proponent 
of  the  view  that  management  refuses  to  take  seriously  well  done  research  in 
the  management  field.  1 certainlv  learned  once  again  the  lesson  that  one's 
own  literature  is  the  best  source  of  criticism.  Thus,  if  researchers  want  to 
feel  badly  about  their  role  in  the  missing  link  bit,  they  merelv  have  to  turn  to 
sources  such  as  tlie  .American  Education  Research  Association's  Rev  let?  of  the 
Educational  Research,  Ptu  Delta  K.ippa's  Kappan , and  the  like.  Conversely,  the 
management  literature  produces  gems  designed  to  make  researcliers  feel  good  (if 
not  full  of  self-pity). 

Since  I enjoy  feeling  good,  I picked  up  a copy  of  the  Industrial  Research 
Institute's  Research  ^lanagement  and  found  a convincing  position  I would  commend 
to  you  at  this  time. 

Stanlev  I.  Berman,  Vice  President  of  Manufacturing  and  Abrasive  Operations 
for  the  Norton  Company,  in  the  .lulv  1973  issue  of  Research  Management  has  an 
article,  the  title  of  which  initially  intimidated  me  (as  a researcher)  consider- 
ably. It  is  "Integrating  the  R&D  Denartment  into  the  Business  Team."  'ir. 
Berman,  not  at  all  abrasive  (as  it  turned  out)  contends  that  management  has 
continually  not  been  able  to  concentual 1 ze  the  need  and  value  of  research.  He 
claims  it  is  the  fault  of  management  that  there  is  a breakdown  in  the  production 
team.  He  states:  "... .management ....  traditionally  has  not  knotvn  what  to  expect 

from  researchers,  and  accordingly  has  given  them  little  direction." 

He  goes  on  to  say  that  management  needs  to  tell  researchers  in  detail 
what  it  requires,  evaluate  the  researchers  capabilities  to  accomplish  the 
objectives  and  then  " if  feasible,  require  them  to  do  it." 

Mr.  Berman  notes  that  management  has  not  supported  research  because  it 
has  traditionally  not  participated  in  R&D,  and  has  not  understood  hov  it 
can  support  R&D  objectives.  He  contends  that  there  needs  to  be  the.  concept 
of  integrating  researcli  and  development  into  total  corporate  allocations — the 
business  plan,  in  effect.  And,  further,  the  R&D  organization  needs  to  be  pro- 
vided with  a careful  definition  of  the  problem  faced  and  the  solutions  required. 
Let  me  quote  Berman  again. 

"If  we  don't  do  this  (providing  the  careful  definition  .and 
the  solutions  required),  if  we  continue  to  allow  a situation 
to  exist  where  we  are  constantly  wondering  how  much  support 
we  are  giving  R&D,  then  we  are  openly  admitting  that  we  as 
managers  have  not  succeeded  in  making  research  an  integral 
part  of  the  business  svstem  and  we  are  risking  total  business 
fail uro . " 
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So,  you  see,  the  solution  to  the  problem  is  clearly  in  the  hands  of 
management . 

Now,  being  in  research  I cannot  help  but  come  forward  with  not  one  but 
several  approaches  for  solution. 

The  problem  is  not  easy  to  solve;  there  are  many  levels  of  competencies 
as  well  as  many  specialties  in  the  research  function  just  as,  in  production 
facility,  one  will  find  a variety  of  competencies  and  specialties  within  the 
work,  force.  In  the  research  function,  for  example,  there  are  personnel  who 
work  well  in  fundamental  research  problems  but  who,  perhaps,  are  impractical 
in  the  broad  field  of  applied  research.  The  converse  is  axiomatic.  The 
Department  of  Defense  RDT&E  structure  speaks  to  such  specialties  in  the  function- 
al terms  of : 

6.1  Fundamental  research 

6.2  Exploratory  development 

6 . 3 Advanced  development 

6. A Engineering  development 

6.5  Management  and  support,  and 

6.6  Operational  Systems  development, 

recognizing  the  unique  facets  of  research  and  development.  It  seems  to  me 
that  management  should  recognize,  if  not  the  categories,  then  certainly  the 
functions  of  the  several  types  of  research  programs. 

Glenn  Bryan,  in  a published  article  entitled  "The  Role  of  Basic  Research 
in  the  Total  R&D  Process",  in  the  January  1973  issue  of  Research  Management, 
conceptualizes  a series  of  models  which  address  the  enigma  of  placement  of  the 
various  types  of  research  into  an  organization. 

Among  the  models  are  several  organizational  ones  with  which  we  are 
familiar.  First,  he  depicts  a "Linear  Production  Model"  which,  for  my 
purposes  today,  adapts  the  DOD  research  functions  into  a production  scheme. 

This  symbol  ( : — ) clijfiotf-s  flow  of  information,  and  forocasts, 

on  "D  .m.in<r' — v.hat  is  worth  tioir.rj  if  science 
/ \ and  tcchnolo'jY  can  provide  the  means  to  do  it 

/ \ 

/ \ 

/ \ 


V / 

/ 


* r 

This  ; I ( »■  I denotes  flow  of  information, 

an.'l  tnre.ea-.ts.  on  "Supply"  — on  what  is  predicted  to 
hi.  tichnoloyically  i .^ll)le, 
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You  can  see,  the  block  to  the  right  draws  upon  and  feeds  responses  to  the 
several  research  functions  to  the  left.  This  situation  is  somewhat  idealized. 
It  appears  in  the  straightforward  approach  that  flow  to  and  from  the  blocks 
is  continuous,  unimpeded,  and  simple.  It  would  be  nice  were  that  so. 

Another  model  of  Bryan’s  is  the  "Departmental  Model." 


oi  Ts*''  '''ni  r/ 


This  model  has  decentralized  the  research  function  and  superior  authorltv 
which  Bryan  calls  "Management"  in  the  model  takes  total  responsibilitv  for 
coordinating  the  several  research  functions  and  moderating  communication  among 
them.  Bryan  states  that  such  a system  as  depicted  in  the  model  requires 
superior  technical  competence  at  the  management  level  to  function  effectively. 
I might  also  add  that  it  will  take  superior  managerial  competence  to  keep  such 
a competitive  system  functioning  toward  the  overall  goals  of  management. 

Another  model  is  the  "Project  Model"  which  has  had  much  usage  throughout 
the  Defense  establishment. 
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The  project  manager  Is  supreme  In  this  model.  He  buvs  what  he  needs 
and  what  he  needs  is  largely  a function  of  his  perception  in  achieving  pro 
ject  goals.  Bryan  points  out  that  such  a structure  has  both  good  and  bad 
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merits.  Inasmuch  as  the  highly  focused  research  efforts,  that  are  in  line 
with  project  goals,  are  often  well  supported  since  they  are  usually  an  adjunct 
to  a high-vislbilitv , large  project,  that  is  good.  The  drawback,  of  course, 
is  that  there  is  little  general  support  for  research  not  directly  in  evlder  * 
of  project  goals.  Research  runs  the  constant  risk  of  being  a necessary  evij. 
in  this  model. 

I won't  show  you  Bryan's  "Organic  Model."  There  is  little  need  to,  for 
consider  a tree,  if  you  will,  laden  with  lush  fruits.  Then  consider  its  root 
structure,  seemingly  disorganized,  ugly,  and  dirty.  Then  consider  the  roots 
as  what  they  are,  the  source  of  the  healthy  fruit;  the  roots  that  are  often 
misunderstood  as  to  their  function  and  their  problems.  That  the  roots  can 
isolate  the  proper  nutrients  that  are  necessary  for  effective  fruit  production 
is  without  question.  Also  without  question  is  the  fact  that  Indiscriminate 
root  pruning  will  almost  certainly  have  adverse  effect  upon  the  fruit. 

Bryan's  tree  is  more  an  analogy  than  a model.  It  is  too  simple  to  portray 
in  any  real  sense  of  the  world  what  the  interaction  between  management  and 
research  is;  it  merely  portrays  what  it  should  be. 

And  this,  perhaps,  is  the  reason  that  we  live  in  a world  of  linears, 
departmental,  and  project  models — variations,  perhaps,  of  those  described. 

A reasoned  approach  by  management  is  necessary  to  bring  about  precisely  that 
fine-grained  relationship  of  mutual  respect  within  organizations  among  its 
major  functionaries.  The  cartoon  showing  the  manager  in  his  plush  office 
speaking  to  a person  in  a laboratory  coat,  and  bearing  the  caption:  "You 

researchers — make  some  kind  of  a breakthrough.'"  has  got  to  cease  being  funny. 

It  would  seem  to  me  that  the  understanding  of  the  R&D  function,  the 
placement  of  that  function  within  the  dynamics  of  the  organization,  and  the 
understanding  by  management  that  reasoned  direction  is  required  will  bring 
the  closure  we  all  seek  and  eliminate  the  need  to  worry  about  the  translation 
problems  between  training  research  and  training  actions. 


SYMPOSIUM 


TRANSLATION  OF  TRAINING  RESEARCH  INTO  TRAI!!ING 
ACTION  — A MISSING  LINK 


DISCUSSION 

Lt  Col  Donald  F.  Mead 
Chief,  Traininc;  Technology  Division 
Randolph  Air  Force  Rase,  Texas 


My  charter  this  morninr  is  to  serve  as  a discussant  of 
the  sonewhat  diverse  views  expressed  by  Dr.  Kerr, 
represent intr  the  trainin;'"  researchers,  an:  Mr.  McDov/ell  , 
the  training  manager.  Following  Dr.  Mayo ' s rr'cceJent  I 
choose  to  exercise  the  pi’erorative  of  1 nt  er.i  ect  i nr  per- 
sonal opinions  when  I feel  they  night  .'.tinulate  thouriC 
or  possible  solutions  to  the  problems  encounterel  in 
the  transition  of  research  data  into  training  prorrams. 
To  assist  you  in  weighing  the  value  of  ny  comments,  I 
might  point  out  that  through  military  assignments  and 
promotions  I have  progressed  through  the  role  of 
researcher  into  that  of  training  research  manager.  Let 
me  say  that  there  is  more  than  a "ju.st  noticeable  dif- 
ference" in  the  two  environments. 

After  a careful  review  of  the  presentations  by  Dr.  Kerr 
and  Mr.  McDov/ell  it  appears  that  the  purposes  of  this 
seminar  could  best  be  served  by  focusing  on  their 
agreements  rather  than  disagreements.  Following  this 
format,  I will  also  discuss  my  personal  reactions  as 
they  occur. 

Of  prime  signficance  to  the  military  services  is  the 
agreement  that  a problem  does  exist  in  the  transition  of 
research  products  into  training  programs.  ""his  should 
trouble  managers  at  all  eciielons  since  it  Indicates 
significant  waste  in  personnel  and  financial  resources. 
Impending  reductions  in  DOD  manning  and  funding  v;ill 
make  this  situation  increasingly  intolerable. 

Vlhlle  our  speakers  agree  that  a transition  problem  does 
exist  their  perspective  on  the  cause  differs. 
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Dr.  Kerr  sees  management  as  the  culprit  due  to  a skewed 
power  structure  and  certain  training  manap-ers  who  are 
inept,  lazy,  and  ignorant.  On  occasion  these  managers 
ignore  irrefutable  research  data,  are  suspicious  of  well 
intended  advice  or  assistance,  and  fall  to  provide  the 
researcher  with  direction  in  the  form  of  a lone-  rane:e 
requirement  plan.  In  view  of  Dr.  Kerr's  description, 
one  has  to  question  his  seeking  direction  from  such  a 
group . 


Dr.  Mavo  concurs  that  manarers  contribute  to  the  problem 
and  sugrests  that  managers  are  generallv  not  receptive 
to  research  due  to  Job  pressure,  disruption  of  operations, 
additional  impact  on  the  staff,  and  failure  to  identify 
with  the  effort.  He  does,  however,  place  part  of  the 
problem  at  the  doorstep  of  the  researcher.  Specii Ically , 
he  cites  the  scientist  who  conceives  of  a problem  area, 
conducts  a study,  then  seeks  out  a training  situation  for 
application.  I think  we  are  all  familiar  with  this 
researcher  v/hom  we  might  liken  to  a free  floating  umbili- 
cal cord  seeking  a v/arm,  receptive  training  course  to 
plug  into  and  thus  nourish  his  creation.  T^hls  fellow 
Is  only  outdone  by  the  research  jockey  who  develops  a 
research  horse  and  rides  it  to  death  from,  iterative  race 
to  iterative  race  with  little  thought  given  to  solving 
problems  and  meeting  needs.  The  prime  objective  here 
appears  to  be  another  Journal  article  or  ticket  to  an 
upcoming  conference  or  symposium. 


Dr.  Kerr  suggests  what  I perceive  to  be  the  real  key  to 
moving  research  products  from  the  laboratory  to  the 
classroom,  and  that  is  comprehensive  training  research 
plans  and  requli’ements . T believe  the  managers'  suspi- 
cion, reluctance,  and  resistance  may  frequent' v be 
traced  to  research  conceived  in  the  isolation  of  a 
laboratory  and  suhseauently  offered  to  solve  a training 
problem.  Typically,  this  requires  the  original  research 
product  to  be  modified,  adjusted,  and  exnanied,  with 
P'redictable  results.  It  is  not  surprising  that  book 
shelves  are  loaded  with  research  reports  never  applied. 
In  many  instances,  I suspect  it  is  because  they  were  not 
designed  in  response  to  stated  operational  problems. 

Along  the  same  lines  but  somewhat  deeper,  I feel  that 
major  modifications  must  be  made  to  gain  coordinated 
management  of  the  multiple  DOD  research  agencies. 
Currently  6.1,  6.2,  and  6.1  efforts  are  often  initiated 
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without  cross-validation  or  coordination  v;lth  training; 
managers.  It  Is  not  unheard  of  to  have  a contractor, 
whose  unsolicited  proposal  Is  funded  with  6.1  monies, 
arrive  at  a military  Installation  without  announcement 
or  coordination.  Similarly,  there  are  examoles  of 
duplicated  efforts  while  topics  needing  exploratory 
research  remain  untouched.  At  this  time  the  military 
services  are  open  to  criticism  In  this  area.  Personally 
I believe  that  no  research,  regardless  of  functional 
category  6.1,  6.2,  or  6.3  should  be  undertaken  unless 
It  supports  stated  user  requirements.  Recently  an 
Interservice  Training  Review  Board  has  been  established 
which  could  make  significant  contributions  In  this  area. 

While  I agree  with  Dr.  Kerr  that  a training  research 
requirement  document  Is  an  essential  Item,  1 ion'*"  believe 
that  It  Is  solely  a management  responsibility.  Ileal Iv 
this  document  would  be  developed  In  coordination  vn*h 
training  researchers.  Carried  a sten  further,  it  cou;  i 
be  a matrix  consisting  of  a taxonomy  of  significant 
training  research  areas  on  one  axis  and  research  age>i:’le- 
on  the  other.  When  ongoing  research  efforts  are  rl‘'^t<-r>a^ 
those  areas  not  being  addressed  become  readily  arrarerd  . 

At  this  point,  I fully  endorse  the  close  coordination 
between  researcher,  manager,  and  trainer  presented  tv 
Mr.  McDowell.  Several  points  of  his  nresentatlon  war’-vant 
reemphasls:  A meeting  between  researcher  and  training 

manager  should  be  held  prior  to  Initiation  of  the  researc'-i 
the  statement  of  work  should  he  coordinated  and  approved ; 
proposed  research  products  should  be  reviewed  and  approved 
as  developed;  and  close  interaction  should  be  maintained 
as  the  project  progresses.  This  coordinated  spearhead 
will  open  a pathway  through  management's  anti -research 
field  of  sharpened  fence  posts  described  by  Dr.  Kerr. 

Another  area  of  general  agreement  among  the  presenters 
Is  the  problem  of  placement  of  R&D  activities  in  the 
organizational  structure.  Dr.  Mayo  suggests  the  R»iD 
group  should  report  directly  to  the  technical  dir’ector 
or  commanding  officer.  I concur  in  this  suggestion. 
Placement  in  the  unper  echelons  of  the  organizational 
structure  is  critical  since  it  dictates  the  Interface 
between  the  training  manager  and  researcher.  In  addition, 
it  eases  access  to  top  decision  makers  which  is  essential 
for  support  of  research  projects  requiring  major  expendi- 
tures of  resources  or  having  significant  Impact  on  the 
training  activity. 
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Because  of  several  unique  aspects  a brief  description  of 
Air  Training  Command's  R&D  organizational  structure  misht 
be  of  Interest  to  this  audience.  Tt  should  be  understood 
that  the  responsibility  for  personnel  research  in  the 
Air  Force  has  been  assigned  to  the  Human  Resources 
Laboratory,  Air  Force  Systems  Command.  Research  require- 
ments from  each  major  command  are  forv/arded  to  them  for 
accomplishment . 

Somev.'hat  unusual  is  the  fact  that  the  HQ  ATC  R^D  orranl- 
zation  is  located  under  the  Deputy  Chief  of  Staff  for 
Plans  rather  than  under  the  DCS  for  Technical  Training 
or  DCS  for  Operations,  v/hlch  involves  flight  and  navigator 
training.  This  placement  perm.its  close  interaction  between 
future  command  prograris  and  R&D  requirements,  permits  more 
comiprehensi ve  scanning  of  state-of-the-art  technology  than 
might  be  possible  under  an  agency  responsible  for  sreciflc 
training,  and  provides  access  to  the  commander. 

Another  unique  aspect  is  the  personnel  make-up  of  the  HQ 
R&D  staf^,  v/hich  consists  of  a computer  specialist,  tv;o 
pilot-fligh*t  sim.ulator  specialists,  a navigator  simulator 
specialist,  an  audio-visual  specialist,  and  two  behavioral 
scientists.  F.ach  specialist  is  charged  to  Interact  v;lth 
academic,  civilian,  and  military  agencies  developing 
products  or>  conducting  research  in  his  area  of  responsi- 
bility. Information  concerning  new  technologies  with 
potential  use  in  the  command  is  coordinated  v;lth  the 
appropriate  DCS  and/or  forvv'arded  to  the  approorlate 
training  activity  for  coordination . Where  interest  is 
shov.’n  arrangements  are  made  for  visitations  to  the  research 
agency  or  briefing  of  appropriate  staff  personnel  by  the 
researcher.  Each  specialist  is  also  responsible  for 
initiating  Requests  for  Personnel  Research  in  his  area  of 
interest  to  be  conducted  by  the  Human  Resources  Laboratory. 

Training  Research  Applications  Branches  or  TRABs  have 
been  established  at  each  Technical  Training  Center, 
monitored  by  the  HQ  R&D  group.  The  TRAP  works  for  the 
school  operations  officer  and  has  access  to  the  school 
common  ier.  F.ach  TRAP  has  a civilian  CC-11  educational 
specialist  and  two  military  behavioral  scientists. 

These  organizations  perform  quick  reaction  studies  for 
training  managers,  conduct  application  studies  with 
training  re.searchers  as  llrected  by  the  HQ  R&D  group , 
and  initiate  research  requests  to  be  accomplished  by  HRL. 
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All  TRAB  projects  and  research  requests  are  controlled 
by  the  HQ  R&D  group  to  prevent  duplicated  effort. 

A Research  Review  Board  made  up  of  the  technical  direc- 
tors from  Technical  Training,  Flight  Training,  and  Plans 
has  been  established  to  validate  command  research  requests. 
In  addition,  they  monitor  the  action  taken  on  recommenda- 
tions resulting  from  completed  research  studies.  The 
Research  Review  Board  responds  directly  to  the  Chief  of 
Staff  and  the  Vice  Commander. 

The  final  area  of  agreement  mentioned  by  our  presenters 
deals  with  the  requirement  for  communications  between 
researcher  and  training  manager.  If  a research  plan  Is 
developed  and  there  Is  close  coordination  during  the 
Initiation,  conduct,  and  reporting  of  research,  communi- 
cations problems  are  rarely  experienced. 

I have  come  to  several  conclusions  regarding  communica- 
tions during  the  past  three  years  of  coordinating  tlie 
Interaction  between  researchers  and  high  echelon  decision 
makers.  Some  of  my  conclusions  concur  and  others  differ 
from  the  positions  taken  todav  by  our  speakers.  Mv 
remarks  reflect  my  personal  opinions  only  and  are  speci- 
fically directed  toward  the  Interaction  between  the 
researcher  and  highest  level  decision  makers. 

Researchers  are  their  own  worst  enemies  In  the  transition 
of  research  products  from  the  laboratory  to  the  training 
enylronment . Specifically: 

a.  As  a group  they  are  particularly  ooor  briefers. 

They  generally  arrlye  orepared  to  deliver  a three-hour 
briefing  to  a person  who  has  limited  time  for  discussion. 
Their  briefing  format  usually  follows  that  of  an  AFA 
Journal  article  and  Is  usually  strongly  statistically 
oriented.  Although  experts  on  their  studies,  they  seldom 
address  the  critical  aspects  affecting  the  decision,  l.e., 
the  total  resource  Impact  of  Implementing  their  work.  T 
recommend  that  prior  to  addressing  a decision  maker  you 
work  with  his  staff  to  develop  a complete  package  and  if 
possible  have  appropriate  members  of  his  staff  present 
to  field  "In  house"  questions  you  can't  answer.  Design 
your  presentation  to  answer  the  question,  "What  does  this 
man  need  to  know  to  make  his  decision?" 
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b.  The  publishing  of  selected  bibliographies  and 

reviews  of  literature  may  look  great  on  the  vita  but  it 
tarnishes  the  researcher's  image  amone  managers.  T 
shudder  when  these  docuraisnts  reach  the  headquarters 
because  they  Invariably  precipitate  two  questions:  "Is 

this  what  we  are  spending  our  money  on?"  and  "What  the 
hell  am  I supposed  to  do  v;lth  this  pile  of  Junk?"  This 
is  particularly  damaging  when  expectations  are  high  con- 
cerning a new  project  and  this  tvpe  document  arrives 
representing  the  initial  v;ork  expended.  My  recommendation 
would  be  to  hold  the  document  and  submit  it  at  the  con- 
clusion of  the  project  or  incorporate  the  entries  with 
your  interim  reports. 

c.  The  time-honored  format  of  the  typical  research 
report  may  be  Justified  for  pucllshlng  ease  and  communi- 
cations between  professional  scientists,  but  it  is  ill- 
designed  to  convey  meaningful  information  to  training 
managers.  Again  I would  encourage  you  to  consider  your 
audience  and  write  to  meet  their  needs.  Minimize  the 
discussion  of  your  statistical  design  and  consider 
presenting  this  Information  as  an  appendix  to  the  report. 

I really  recommend  dropping  the  APA  report  format  and 
going  to  executive  summiary  type  reporting. 

d.  And  finally,  v\rhen  a fast  reaction  situation 
requires,  stick  your  neck  out.  If  a decision  has  to  be 
made,  don't  suggest  you  can  provide  a solution  in 

18  months.  Use  your  knowledge  of  research  data,  and 
come  up  with  viable  options.  Validate  as  you  go  and 
modify  the  approach  as  required. 
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SYMPOSIUM 

TRANSLATION  OF  TRAINING  RESEARCH  INTO  TRAINING  ACTION  - A MISSING  LINK 


DISCUSSION  AND  COMMENTARY 
EARL  I.  JONES 

NAVY  PERSONNEL  RESEARCH  AND  DEVELOPMENT  CENTER 
SAN  DIEGO,  CALIEORlilA  92152 


Having  reviewed  with  great  interest  the  papers  of  Mayo,  McDowell  and 
Kerr  I shall  attempt  to  summarize  their  main  points,  comment  upon  them, 
and  then  as  is  a discussant's  privilege,  provide  some  comments  of  my  own. 

Dr.  Mayo,  with  whom  I have  worked  closely  for  the  better  part  of 
two  decades  makes  several  major  points.  His  first  major  point  seems  to 
me  to  be  the  rationale  of  this  symposium.  It  is  that  a great  amount  of 
training  research  and  development  goes  unused.  He  stipulates  that 
failure  to  use  R5D  cannot  be  accounted  for  by  the  simple  assumption  that 
risk  taking  is  a fundamental  part  of  R&D  nor  by  the  fact  that  some  of 
the  unused  training  R&D  is  indeed  unusable.  He  leaves  out  an  important 
(but  fortunately  small)  class,  namely,  training  R&D  that  has  been  used 
but  should  not  have  been. 

The  next  major  point  in  Dr.  Mayo's  paper  is  his  reference  to 
"linear  change  models  in  education"  with  the  stipulation  that  conditions 
of  intrinsic  or  extrinsic  motivation  must  be  functioning  for  the  model 
to  work. 

From  here  Dr.  Mayo  moves  to  the  position  that  most  of  the  conditions 
functioning  to  obviate  use  or  translation  of  training  R&D  are  associated 
with  (1)  research  personnel,  (2)  training  personnel,  or  (3)  the  interface 
or  interaction  between  research  personnel  and  training  personnel.  With 
respect  to  research  personnel  Dr.  Mayo  perceives  the  major  sin  to  be 
conceiving  studies  in  which  researchers  look  f 'r  training  situations  in 
which  to  execute  but  fail  to  consider  (in  tha  planning  stage)  the  utility 
or  implementability  of  the  R6D  results.  As  a solution  to  this  unwhole- 
some state  of  affairs  Dr.  Mayo  has  the  pragmatic  notion  that  researchers 
should  pre-state  and  pre-judge  the  'raining  actions  to  be  taken  contin- 
gent upon  the  possible  R&D  outcomes  and  accept  responsibility  for  such 
planning . 

With  reference  to  training  personnel  Dr.  Mayo  describes  their  work 
overload,  their  dislike  of  waiting  forever  for  utilizable  R&D  results 
or  recommendations,  their  commonly  not  identifying  with  the  R&D  task  or 
project,  and  tlieir  feeling  tfiat  whoever  might  get  credit  for  research 
based  action,  they  (the  training  personnel)  will  not. 
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Dr.  Mayo  sees  the  interface  or  interaction  between  researchers  and 
trainers  to  be  the  most  critical  problem.  }iere  he  recognizes  that  the 
continuity  from  R&D  to  translation  to  change  in  training  can  be  accom- 
plished only  by  appropriate  interaction  in  which  researchers  and  their 
managers  take  the  responsibility  for  the  translation  process  and  assist 
the  training  personnel  in  the  action  process  to  the  point  where  appro- 
priate implementation  has  been  accomplished. 

Mr.  McDowell's  presentation  is  a natural  follow-on  to  Dr.  Mayo's. 

He  perceives  the  requirement  for  proposing  research  as  a soul  searching 
responsibility  for  use  of  the  taxpayer's  dollar  involving  the  questions 
of  benefits  gained  from  past  R6D  and  pr'ojected  yield  of  future  R&D.  He 
perceives  the  trairiers’  and  managers'  roles  as  not  limited  to  translation, 
but  a process  of  bridging  the  gap  bet’ween  conduct  of  research  and  imple- 
mentation. This  process  is  continuous  according  to  Mr.  McDowell  and 
requires  constant  interaction  between  researcher  and  trainer  from  problem 
identification,  Rf-D  planning,  conduct  of  R&D  through  translation  and 
implementation . 

Mr.  McDowell  is  not  so  prone  as  Dr.  Mayo  to  perceive  a vast  grave- 
yard of  unused  R&D.  He  qualifies  his  remarks  in  a way  which  implies 
that  tlie  Army  may  not  suffer  the  "missing  link"  dilemma  to  the  extent 
that  the  liavy  does. 

Where  Dr.  Mayo  exhorts  the  need  for  trainer-researcher  interaction, 
Mr.  McDowell  talks  of  a system  in  being.  He  refers  to  interaction  that 
"takes  place;"  data  which  are  "provided  to  the  trainer;"  reports  which 
are  "reviewed  and  staffed  with  appropriate  elements  of  the  organization;" 
and  specific  examples  of  frequent  and  essential  involvement  of  the  re- 
search team  in  "active  participation."  He  perceives  use  of  R&D  by- 
products as  increasing,  and  dissemination  of  RSD  findings  among  potential 
user.s  as  essential. 

In  closing,  Mr.  McDowell  excerpts  from  Dr.  McClelland's  1967  report 
eight  characteristics  of  unsuccessful  research  efforts  and  five  charac- 
teristics of  successful  research  efforts. 

The  final  presentation  by  Dr.  Norman  Kerr  is  a shocker.  In  some 
communities  it  might  even  seem  blasphemous.  However,  in  the  long  course 
of  the  history  of  education,  the  history  of  training  and  the  history  of 
training  R&D,  Dr.  Kerr's  remarks  spark  a flame  of  truth — albeit  a truth 
so  mixed  with  exception,  complication  and  dark  shadow  that  its  explica- 
tion is  no  simple ‘m.at ter . 

Dr.  Kerr's  paper  states  what  the  other  papers  only  imply.  The 
problem  miay  not  be  a missing  link  but  a faulty  chain  anchored  inse- 
curely and  subject  to  vectors  of  power  and  stress  that  make  the  task 
of  translating  and  applying  R&D  results  in  a positive  manner  unduly 
difficult  or  even  impossible. 
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Dr.  Kerr's  major  points  are  that  training  managers  obviate  rather 
than  facilitate  R&D,  that  they  possess  power  whereas  researchers  do  not, 
and  that  the  power  disparity  at  all  levels  of  Interaction  between 
training  managers  and  training  researchers  are  the  central  focus  for 
consideration  in  this  s^miposium.  Dr.  Kerr  carries  this  argument  with 
conviction,  verve,  eloquence  and  numerous  examples.  Although  he  pro- 
vides a scorching  indictment  of  training  managers.  Dr.  Kerr  does  not 
back  slowly  out  of  the  nearest  exit.  He  finds  to  his  dismay  that  his 
review  of  research  literature  does  not  provide  the  confirmation  he 
seeks  but  that  management  literature  do<^s.  With  support  from  research 
management  literature  and  his  own  knowledge  of  the  R6D  community's 
management  models.  Dr.  Kerr  provides  some  positive  proposals  for 
rapprochement . 

We  now  have  summarlr.ed  the  major  points  made  by.  the  three  speakers. 
Let  us  comment  briefly  upon  them. 

To  my  ear  at  least , the  three  papers  we  fiave  heard  are  extremely 
useful  accounts  of  the  success  or  failure  of  military  organizations  to 
exploit  R&D  as  a means  of  upgrading  military  training.  As  we  might 
expect  the  papers  strongly  reflect  both  the  nature  and  nurture  of  the 
problem  and  the  histories  of  the  presenters.  I,  too,  will  try  to 
overcome  that  dilemma  of  all  ohservers--how  to  get  outside  of  my  own 
skin  while  being  securely  bound  within  it. 

First  for  Dr.  Mayo.  The  explicit  concurrence  of  both  Mr.  McDowell 
(by  his  reference  to  McClelland)  and  Dr.  Kerr  as  a strong  protagonist 
would  be  sufficient  for  me  to  accept  the  assumption  on  which  this  sym- 
posium is  based;  that  is,  that  there  is  a significant  amount  of  useful 
training  R6D  that  does  not  get  exploited  by  the  military  training 
community . 

Time  does  not  permit  complete  comment  on  "linear  change  models;" 
suffice  it  to  say  that  they  are  useful  devices  which  like  most  models 
have  various  constraints.  The  primary  one  is  perhaps  the  need  for 
intrinsic  or  extrinsic  motivation.  Here  Mr.  McDowell's  cautionary 
observation  that  a decree  by  management  (one  type  of  extrinsic  motiva- 
tion) will  not  get  the  job  done  is  fair  warning  and  is  ecfioed  in 
different  ways  in  both  Dr.  McClelland's  list  and  in  Dr.  Kerr's 
admonitions.  L'xtrinsic  motivation  is  tfien  sometimes  necessary  but 
seldom  sufficient.  However,  given  that  one  of  the  motivational 
conditions  is  necessary,  the  problem  becomes  one  of  controlling  the 
direction  of  its  outcome. 

Within  the  context  of  this  symposium.  Dr.  Mayo's  point  that  transla 
tion  of  R&D  into  action  depends  upon  research  personnel,  training  person 
nel  and  their  interaction  is  so  strongly  supported  by  McDowell,  Kerr 


and  logic  that  it  needs  little  conmentary  except  to  wonder  why,  when  the 
problem  can  be  cast  into  such  simple  terms,  the  solution  to  the  problem 
has  not  long  ago  been  routinely  applied. 

Mr.  McDowell's  paper  is  so  convincingly  positive  that  it  would 
seem,  for  the  Army  at  least,  the  solution  is  at  hand. 

Shifting  now  to  Dr.  Kerr  provides  a somewhat  stark  contrast.  He 
ignites  such  a conflagration  that  we  are  heaped  with  ashes  and  wonder 
as  in  the  "Perils  of  Pauline"  if  there  can  ever  be  a happy  ending.  By 
following  Dr.  Kerr  further,  the  most  obvious  answer  would  appear  to  be 
a qualified  "yes."  Equitably  distribute  power,  select  a management 
model,  integrate  R5D  into  the  team  and,  I will  infer,  we  will  be  on  our 
way  to  some  kind  of  break  through--not  ordained  and  not  funny. 

To  me,  a bothersome  aspect  of  Dr.  Kerr's  presentation  is  his 
broken  field  running.  He,  with  some  justification,  puts  the  main  burden 
on  training  managers.  He  finds  researchers  to  be  hushed,  humble  and 
contrite  (hardly  characteristic  of  his  own  presentation)  and  finds 
solace  in  management  literature.  His  two  major  references  are  Berman 
and  Bryan.  Somie  com.ment  is  called  for  here.  Both  of  these  references 
deal  not  just  with  managem.ent  but  with  research  m.anagem.ent . Berman's 
message,  intended  for  the  industrial  world,  is  that  the  R&D  component 
of  a company  should  be  part  of  the  business  team — to  participate  with 
and  interact  with  management,  for  decades  this  has  been  true  of  such 
major  companies  as  Bell  Telephone,  IBM,  Union  Carbide  and  countless 
others.  Dr.  Bryan's  article  was  motivated  by  a three  day  R6D  symposium 
concerned  with  the  adequacy  of  DOD  R&D  structure  and  its  Implied  lineal 
model  for  the  management,  planning,  funding  and  execution  of  manpower 
and  behavioral  science  R6D.  Dr.  Kerr  uses  these  references  to  advan- 
tage in  making  his  sti’ongest  positive  point,  the  need  for  research  and 
managem.ent  organisations  to  become  an  integral  force  for  product  im- 
provement and  organization  efficiency.  Had  Dr.  Kerr's  search  of  research 
literature  been  somewhat  miore  extensive  he  would  have  found  a good  deal 
of  support.  The  writings  of  Finn,  Lum.sdaine,  Glaser,  Rigney,  Hackie,  and 
even  Earl  Jones  v/ould  not  only  be  consoling  but  might  dispel  his  notions 
of  contrite. 

If  my  interpretation  of  these  papers  has  been  correct,  I heartily 
endorse  not  only  their  major  aim  but  their  explication  of  factors  which 
obviate  implementatior,  of  R&D  into  training  action  and  their  recommenda- 
tions for  positive  steps  for  catalyzing,  facilitating  and  improving  the 
translation  and  implementation  process.  Yet  I think  they  have  not  gone 
far  enough.  What  seems  to  me  to  bo  missing  are  a set  of  models  for  training 
management,  R&D  management  and  a meta  model,  superordinate  to  training  and 
R&D,  which  could  guide  both  the  training  community  and  the  R&D  community  in 
the  spirit  of  integration,  establish  all  the  necessary  interfaces  at  all 
levels  of  function  in  both  training  and  training  R&D  and  provide  such  mechanisms 
as  a Training  Research  Advisory  Board  and  laboratory  schools--in  short,  a model 
which  could  facilitate  the  kinds  of  interactions  all  three  of  our  speakers  have 
proposed  in  their  own  viniqvic  styles. 
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INTRODUCTION 


The  public  seems  to  be  generally  willing  to  accept  a decrease  in 
the  manpower  of  the  armed  services.  As  a result  of  this  acceptance  the 

active  draft  system  has  been  deleted  as  a method  of  acquiring  service- 

men. Whether  this  acceptance  is  warranted  remains  to  be  decided  however, 
and  in  the  meantime  this  decrease  in  manpower  is  something  with  which 

the  military  forces  must  dqal.  Why  must  we  deal  with  it?  Because,  in 

the  event  that  military  or  naval  forces  must  be  called  to  .action  on 
short  notice,  the  American  people  will  expect  and  demand  tlie  same 
readiness  and  response  which  has  always  been  characteristic  of  the 
armed  forces.  An  uneffective  response  at  such  a time  could  cause  a 
breakdown  in  one  or  all  of  the  systems  which  defend  our  National 
Security.  It  is  apparent  then  that^in  the  face  of  major  cutbacks  in 
the  quality  and  quantity  of  individuals  joining  the  anned  forces,  we 
must  continue  to  maintain  an  effective  state  of  military  re.adiness 
which  is  responsive  to  situation  demands. 

Many  times  we  lose  track  of  the  overall  importance  of  our  jobs 
when  we  become  involved  in  the  tasks  of  everyday  business.  However 
such  large  problems  as  that  i-ntnitioned  above  are  iiot  renunlied  by  one  or 
even  several  decisions  or'  changes.  The  overall  situation,  in  this 
case  is  affected  by  many  imall  variables  and  'ution  to  it  will 

be  dependent  upon  many  small  adjustments  (ir  changes  .hich  will  in 
turn  have  an  overall  effect. 


The  situation  with  which  this  study  de.ils  is  one  of  those  small 


areas  In  which  changes  can  be  made  which  will  affect  the  ability  and 
readiness  of  the  men  presently  being  trained  by  our  armed  forces. 

To  increase  the  abilities  of  our  servicemen  in  their  various 
specialties,  as  well  as  in  the  general  knowledge  of  those  things  which 
pertain  to  the  military  forces,  is  an  objective  v;hich  will  help 
accomplish  the  readiness  which  the  public  expects.  This  can  be 
achieved  even  in  the  face  of  lower  enlistment  rates  and  manpower 
cutbacks.  In  increasing  retention  and  usage  of  information  given  to 
trainees  we  can  project  that  this  ability  will  enable  them  to  operate 
at  an  increased  level  of  awareness,  and  efficency. 

At  the  Coast  Guard  Training  Center  in  Cape  May  New  Jersey  we  often 
encourage  instructors  to  give  information  to  recruits  in  regard  to  per- 
formance both  on  the  positive  as  well  as  negative  side.  This  feedback 
is  what  most  people  base  their  evaluations  of  their  abilities  in  any 
particular  area  on. 

In  an  early  experiment  on  feedback  Hurlock  (1925)  emphasized 
encouragement  or  incentives  and  their  effect  on  learning.  He  also 
encouraged  the  use  of  verbal  praise  or  criticism  and  added  power  to  the 
preference  for  verbal  as  opposed  to  other  types  of  feedback.  Plowman 
and  Stroud  (19^2)  showed  that  seeing  their  errors  corrected  on  an 
examination  allowed  high  school  students  to  eliminate  approximately  50^ 
of  their  errors  on  a retest  one  week  later.  This  result  was  probably 
due  a great  deal  to  the  practice  effect  but  does  point  to  the  value  of 
feedback  In  enhancing  ability.  More  recently  Page  (1958)  reported 


results  which  bear  out  Hurlock's  earlier  emphasis  on  praise  and  blame. 

He  found  that  students  who  received  comments  on  examination  papers  did 
better  on  the  next  examination  then  did  students  who  received  no  coiments 

Much  of  the  previous  research  in  this  area  has  been  conducted 
comparing  feedback  with  no  feedback  situations.  Curtis  and  Wood  (1929) 
however,  tested  four  methods  of  scoring  examinations  and  found  that 
those  methods  which  provided  for  discussion  of  questions  were  the  most 
effective.  Stone  (1955)  found  that  the  effect  of  feedback  was  correlated 
positively  with  the  amount  of  information  contained  in  the  feedback. 
Students  who  received  full  explanat ion, as  to  why  one  answer  was  v;rong 
and  another  was  correct,  did  much  better  subsequently  than  did  those  who 
received  less,  or  no,  information.  Sassenrath  and  Carverick  (1965) 
compared  four  treatment  groups  for  retention  and  transfer  of  learning 
and  concluded  that  the  type  of  feedback  is  not  nearly  as  important  as 
the  fact  that  the  group  gets  it.  In  the  present  experiment  it  was 
hypothesized  that  the  introduction  of  weekly  quizzes  prior  to  the  first 
progress  test  in  recruit  training  would  increase  scores  on  the  progress 
test  and  this  increase  would  vary  positively  in  accordance  with  the 
amount  of  feedback  given  on  these  quizzes. 


METHOD 


APPARATUS  Quizzes  consisting  of  twenty  five  questions  each,  based  on 
the  first  and  second  week  subject  matter  were  constructed.  Questions 
covered  the  same  subject  areas  as  those  on  the  third  week  progress  test 
but  were  changed  to  appear  in  a slightly  different  form. 
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All  quizzes  were  given  in  well  lighted  classrooms  under 
comfortable  conditions. 

SUBJECTS  and  DESIGN  Five  hundred  and  sixty  eight  Coast  Guard  recruits 
who  entered  the  service  between  15  April  and  15  August  1973  served 
as  subjects  in  this  experiment.  They  were  assigned  to  companies, 
approximately  fifty  men  each,  as  they  reported  for  duty.  The  varying 
numbers  in  each  company  caused  the  number  of  subjects  in  each  treatment 
group  to  fluctuate.  As  each  company  of  men  began  training  they  were 
assigned  a treatment  category,  1,  2,  3.  or  A.  This  continued  until  four 
companies  had  gone  through  treatment  1,  four  companies  had  treatment 
2,  five  companies  had  treatment  3,  and  four  companies  had  treatment  A. 
The  total  number  of  subjects  in  each  group  was  as  follows. 


Treatment  ^1  - lA5 

Treatment  HI  - 13^ 

Treatment  ?3  “ 170 

Treatment  - 119 


568  = N 

Subjects  were  therefore  assigned  as  randomly  as  possible  to  treatment 
groups.  Four  instructors  took  turns  administering  the  three  treatments 
to  help  control  for  the  effect  of  any  particular  instructor  personality. 

PROCEDURE  Prior  to  this  time  no  quizzes  were  given  to  recruits  other 
than  the  third  and  sixth  week  progress  tests  which  consist  of  50 
multiple  choice  questions  on  material  covered  in  lecture,  course  material 
and  text.  Quizzes  were  introduced  at  the  end  of  weeks  one  and  two  prior 
to  the  first  progress  test.  Companies  received  their  assigned  feedback 
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condition  over  both  quizzes  and  were  afterwards  tested  using  the  standard 
"first  progress  test".  The  treatments  for  each  group  are  specified  below. 
Recruits  were  told  that  quiz  grades  would  not  be  a part  of  their  record 
and  were  for  their  information  only. 

TREATMENT  #1 

After  the  quiz  each  man  kept  his  paper  and  the  answers  were  reviewed. 
When  no  questions  came  up  in  regard  to  an  item  the  instructor  was  told  to 
explain  the  answer  anyway.  Recruits  were  encouraged  to  ask  questions  and 
discuss  each  one  fully. 

TREATMENT  #2 

The  companies  which  took  part  in  this  treatment  were  told  which 
choice  was  correct  in  each  case,  but  were  not  given  any  further  information 
or  allowed  to  question  it.  Thus  they  received  some  feedback  but  not  the 
amount  available  to  treatment  group  one. 

TREATMENT  #3 

Since  we  were  introducing  two  independent  variables  in  the  feedback 
and  the  quiz  itself,  it  was  necessary  to  be  able  to  say  wether  change 
was  due  to  the  feedback  or  merely  the  introduction  of  the  quiz.  It  was 
for  this  reason  that  group  three  received  only  the  quiz  and  no  information 
as  to  hov/  well  they  did  on  it.  Instructors  required  that  answer  sheets 
be  turned  in  upon  completion  and  gave  no  information  on  correct  answers. 
Subjects  received  no  feedback  other  than  how  they  felt  they  had  done  on 
the  quiz. 

TREATMENT  §k 

The  control  groups  proceeded  through  training  as  other  companies  had 
until  this  time,  with  no  quizzes  until  the  progress  test  at  the  end  of 
the  third  week  of  training. 
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RESULTS 


A Kruskal  - Wallis  one  way  analysis  of  variance  yielded  an  He  value 
significant  at  above  the  .05  level.  Group  comparisons,  again  using  the 
Kruskal“Wal 1 i s formula,  were  made  which  indicated  significant  differences 
between  several  of  the  specific  groups. 


INSERT  FIGURE  1 HERE 


These  results  imply  that  a definite  difference  existed  between  the 
group  which  received  maximum  feedback  from  the  quizzes  and  both  the 
control  and  the  quiz  - no  feedback  group.  This  difference  bears  out 
the  hypothesis  and  was  presumably  due  to  the  effects  of  feedback.  The 
max  feedback  group  did  not  however,  differ  significantly  from  the 
minimum  feedback  group  (group  ll).  In  this  respect  the  increase  in 
feedback  did  not  cause  a significant  difference  from  the  lesser  feed- 
back level  and  falls  to  bear  out  the  finer  implications  of  the 
hypothesis.  The  trend  was  toward  an  increase  in  feedback  predicting 
increase  in  performance,  but  in  this  case  it  did  not  approach 
s ign 1 f icance . 

DISCUSSION 

The  results  of  this  experiment  bear  out  the  hypothesis  in  regard 
to  feedback  versus  no  feedback  and  seem  to  confirm  a statement  made 
previously  (Sassenrath  and  Garverick,  1965)  that  the  type  or  amount  of 
feedback  Is  not  of  extreme  importance  in  these  situations.  The 
important  thing  is  that  there  is  some  feedback.  The  data  shows  a 
trend,  although  not  always  at  significant  levels,  that  as  the  amount 

of  feedback  increases  so  does  performance. 
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Instructors  noted  that  throughout  the  experiment  the  groups  which 
received  max  feedback  showed  a good  deal  of  interest  in  other  areas  of 
training  and  asked  questions  which  concerned  more  than  just  the  quiz. 

One  important  factor  in  this  case  seems  to  be  that  of  human  interaction; 
the  chance  to  speak  with,  or  question,  someone  higher  up  in  the  chain 
of  command.  This  interaction  was  seen  to  have  an  effect  in  terms  of 
heightened  interest  in  many  areas  of  military  life.  These  companies 
received  the  treatments  twice  over  a two  week  period  and  the  question 
crops  up  wether  the  difference  in  group  one's  performance  over  that  of 
three  and  four  was  due  to  the  educational  aspects  of  feedback  or  the 
psychological  aspect  of  interaction  with  someone  of  higher  rank  on  a 
semi  informal  basis.  Both  factors  probably  affected  the  results  but  it 
would  be  interesting  to  see  wether  the  feedback  was  valuable  because  of 
its  information  or  because  of  the  human  interaction. 

A certain  type  of  practice  effect  probably  occurred  also  in  that  a 
majority  of  these  men  indicate  a fear  or  apprehension  concerning  test 
taking.  Allowing  them  to  take  two  quizzes  prior  to  the  progress  test 
probably  helped  eliminate  some  of  the  anxiety  normally  surrounding 
this  test  and  present  in  group  four. 

Due  to  the  nature  of  the  environment  subjects  in  this  experiment 
could  not  be  assigned  to  treatment  groups  on  a completely  random  basis. 

There  is  a fluctuation  in  recruit  education  level  that  occurs  at  various 
times  throughout  the  year^and  it  is  possible  that  this  fluctuation  may 
have  had  some  effect  on  results.  However  companies  were  assigned  a 
treatment  group  in  sequence  as  they  entered  training  and  this  fact  should 
have  ramdomized  the  effects  of  the  seasonal  fluctuations. 

Whatever  the  reasons  for  the  effects  of  feedback  in  various  situations. 
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we  know  that  it  does  enhance  performance  and  increases  retention  of 
information  to  a recognizable  extent.  This  realization  should  be  of 
prime  importance  in  developing  any  training  program  and  should  incoroorate 
the  use  of  human  interaction  in  increasing  a persons  interest  and 
knowledge  in  their  chosen  area  of  service.  Feedback  allows  individuals 
to  assume  their  Identity  as  humans  with  individual  scores,  performances, 
abilities  and  characteristics.  Feedback  is  the  basis  of  how  we  feel 
about  ourselves  and  we  rely  on  it  to  continue  shaping  our  ideas.  The 
more  of  it  we  get,  the  better  we  are  able  to  evaluate  our  abilities. 

In  a society  which  deals  in  numbers,  feedback  is  a chance  to  conserve 
our  identity  and  enjoy  our  work.  As  such  it  should  be  at  the  heart  of 
any  training  program. 
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RESULTS 


OVERALL  SIGNIFICANCE  AT  BETTER  THAN  .05  LEVEL. 
REQUIRED  7.82  OBTAINED  H = 8.19 

AFTER  THE  CORRECTION  He  = 8.21 

GROUP  COMPARISONS 


H 


vs 

1 1 

LESS  THAN  . 

05  LEVEL  - 

2.51 

vs 

1 1 1 

HIGHER  THAN 

.01  LEVEL- 

10.15 

vs 

IV 

HIGHER  THAN 

.01  LEVEL- 

23.3^ 

FIGURE  1 
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"PROBE"'  An  automated  testim  system  for  +te  evaluation 
of  ability  and/or  willinaness  to  profit  ■‘rom  a 
+rainina  situation 

David  B.  '/‘nsop,  pa.D. 

I7|0  ''edical  Towers  Rldn. 

Houston  Texas 


In  COG,  a computer  software  nrooram  capable  of 

assessina  the  effect  of  cerebral  concussion  on  connitive 
function  was  reported.  Us  i no  repression  analvsis,  POro 
compared  baseline  data  with  post-iniurv  performance  data 
to  calculate  the  probabilitv  of  impaired  connitive  function 
associated  with  cerebral  concussion. 

Puitdinn  on  experience  nained  with  roo,  a co'"puter 
software  pronram  op ORE  was  de\eloped  to  evaluate  certain 
connitive  and  personality  functions,  and  -^rom  the  inter- 
action of  these  functions  as  specified  hv  alnorithm, s,  to 
predict  the  subiect’s  ability  and/or  willinnness  to  profit 
from  a traininq  pronram. 

In  the  PPOPf  system,  performance  data  are  acnuired  bv 
the  administration  of  +ests  ti-'  assess  selective  attention, 
short-+erm  memory,  rate  of  inductive  reasoninn,  s^eed  and 
accuracy  of  continuous  addition,  level  of  asriration,  ability 
to  read,  and  mental  attitudes.  Pertain  bionranhical  data  are 
input  to  the  system,  but  sex  and  culture  are  not  computed. 

Written  in  Porfran  IV.  +he  opopc  software  uses  approxi- 
mately 27  K characters  with  two  overlays  of  ll'*'  and 
respectively.  Csinn  an  XDS-'^4o  |p  tim^-jhare,  performance 
data  are  input  and  comip, ared  with  stored  informat’on  to 
nenerate  a report  which  nuantitativelv  nrades  a subiect  on 
five  ■‘actors  previously  identified  bv  factor  analysis  as 
predictive  of  supervisor  ratines  o+  iob  performance. 

‘‘he  five  factors  as  de'^ined  in  the  opopp  print-out 
reports  are  mental  arasn,  self-confidence,  coon e r a t i ve nes s , 
emotional  stability,  and  leadership.  npopp  nradinns  are 
from  1.0  to  b.O,  reported  to  one  decimal  point.  A nradino 
of  3.0  represents  the  mean  of  hinh  performinn  normal  subiects. 
poq^E  has  been  on-line  since  I'^'^l.  '"ron  receipt  data,  +o 
user's  receipt  of  the  report,  the  turn  around  +ime  is  less 
than  ten  minutes  ner  subiect.  anvv/here  in  " !r  + h America. 
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The  svstem  has  the  advantaee  of  usinn  non-nrofessionals 
for  the  acouisition  of  hioqraphica!  data  and  for  the  adminis- 
tration of  the  sub  tests  of  coanitive  and  personal  itv  functions, 
‘fost  subtests  are  paper-and-penc  i I : the  short-term  memory 
subtest  is  automa  + ed.  h'ith  the  exception  of  one  tes+,  the 
Color  Word  test,  the  battery  is  appropriate  for  uroup  adminis- 
tration and  the  subject  records  his  ov.-n  responses  on  an  answer 
sheet.  For  the  Color  Word  test,  ff’e  adm  i n i stra  tor  records  the 
responses.  Performance  data  m-jy  be  transmitted  to  the  seryice 
bureau  for  processinn  bv  any  of  several  telecommunication  mode'". 

The  data  ta  e for  the  noonp  '-cftware  is  in  -*-he  m^nv 
thousands  of  suriects  and  includes  the  so-called  normals 
(those  who  .inplv  for  iors,  demonstrate  their  co"'petence  hy 
earnina  promotions,  provide  for  their  own  needs,  and  for  the 
needs  of  their  dependents).  To  establish  limits  of  performance, 
other  classes  of  suticcts  were  innut  to  the  data  hace:  patients 

of  Known  psvchia+ric  or  neuroionic  diannoses,  adolescents, 
the  aned,  stress  - "eei-er',  compe+itive  athletes,  and  several 

classes  of  the  so-called  unemrlovahle. 

» 

The  PPPipr  rep'.-'rt  sample  is  shov.n  as  ‘‘inure  I.  |f  desired, 
values  of  the  five  parameter-  can  he  nraphicallv  displayed  (bv 
hand  or  bv  if-''  plotter)  as  shown  in  the  d tarr-'^lot,  ‘‘inure  . 

The  plottinn  points  of  the  dtarr-Plot  represent  performance  as 
it  falls  about  the  me')'-,  where  t.O  represents  the  mean, 
one  standard  deviation  above  t‘ie  mean,  d.ia  one  standard  rjeviation 
below  the  e*c. 

‘‘’POTF  is  desinned  to  yield  a report  for  use  in  worK  and/or 
trainim  situations.  ‘‘ron  the  same  performance  da^+a  which  have 
been  acnuired  for  FPORf , a opv  report  can  he  nenerated  which  is 
desinned  for  use  in  medical  evaluation  o'  treatment  situations. 

A POy  repr'"T,  shown  in  ‘‘inure  ^ , is  ha'^ed  on  the  same  nerfnrmance 
da  fa  used  to  non  er  ate  the  pf^Fipp  renor-f  in  ‘^inure  I. 

Comparisons  between  predicted  and  observed  ahllltv  and/or 
wlllinnness  to  profit  from  a tr a ininn  nronram  have  been  made 
extensively.  ^ou  r formal  validations  of  ooonr  ar<">  summarised 
as  Studies  A throunh  D . 

STUDY  A;  Five  professional  football  teams  each  +ested 
seventy  players.  ‘'s  the  external  criterion,  nlavers  wepo 
nraded  bv  coach i nq  staff  members  who  had  observed  the  nlayer 
at  least  one  year.  By  correlation  ena  lysis,  these  ora d inns 
were  compared  with  ‘■’OORF  nradinns.  (A|-fhounh  the  lannuaoe  of 
the  football  reports  is  Keyed  to  sport,  the  alnorithms  which 
predict  ability  and/or  willinnness  to  profit  from  a traininn 
prooram  are  essentially  the  samo  as  those  of  ooopr,) 
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Althouqh  supervisor's  rations  of  wori'  performance  are 
aenera I Iv  acknowledged  to  be  biased  toward  the  mean,  oradinos 
of  professional  football  coaches  are  not.  Coaches'  nradinns 
of  certain  observable  behavior  associated  with  on-the-iob 
performance  are  as  shown  in  Table  I. 
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TEAM 

LEARMI mc 

COUr 1 PENCE 

CO  ACM/'  R 1 L 1 TY 

fl  r,  0 D r c;  c;  1 V F ► 1 F h S 

A 

. 0 1 

.0  1 

. Oh 

. o 1 

B 

.Oh 

. I 0 

. 0 1 

. 0 1 

C 

. 00  1 

n s 

.00  1 

n s 

n 

. 0 1 

. 1 0 

. 1 p 

. os 

E 

.02 

.Oh 

. OS 

n s 

STUDIES  B-C: 

Performance 

da+a  were  acnu 

i red  on  To  wh  i 

(Ftudv  P)  and  A3  N'enro  (Studv  C)  telephonf^  operator  trainees 
to  test  the  null  hypothesis  that  P'^ORE  oradinos  would  di^'^er 
+ ron  supervisor  ratinqs,  observed  behavior,  and/or  nerfor'^ance 
data  to  an  extent  no  ore a ter  than  chance. 

Subiects  to  Studies  B and  C were  total  Iv  unselected,  that 
is,  each  applicant  for  work  in  a niven  time  period  was  hired, 
tested  and  trained  bv  the  companv  unless  she  was  excluded  hv 
medical  findinos  of  tuberculosis  or  venereal  disease.  Subiects 
were  assessed  bv  pRPBE  within  hours  of  renortino  to  work  for 
the  first  time.  After  each  employee  had  been  on  +he  ioh  for 
days,  each  was  subiectivelv  rated  bv  supervisors  to  whom 
PP'^iRE  reports  were  not  avaiiable. 
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s and  the 
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pf  POORF 
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the  aoreenent 

is  as  shown  in  Table  ?. 
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^ or  Study  C,  the  agreement  is  as  shown  in  T.ihle 

TAR|  F S 


M e n t a 1 

Cnn  f i 

Poope  ra- 

S t a h - 

leader- 

Grasp 

d en  CP 

t i V en  e s 5 

i 1 i +v 

ship 

Know  1 edge  of 

. on  1 

n s 

n s 

. OS 

. no  1 

Product i V i tv 

. no  1 

n 5 

n s 

n s 

. oo  1 

Pr i de  in  Work 

. 07 

.0^ 

. no  1 

.00  1 

. oo  1 

Learnina  Rate 

.00  1 

n s 

n s 

n s 

. 0 o 1 

Over  a 1 1 

.00  1 

n s 

n s 

n s 

. on  1 

F rom  the  i 

n f o r m a f 

ion  in 

Tables  R and 

3 , it 

appeared 

the  superv i sors 

( 1 ) d i 

d -find 

the  <^ive  parameters 

of  PRopr 

relevant  to  observed  job  nerformance:  ("’)  saw  the  two  races 

differently:  and  (3)  nraded  ''oloballv,"  that  is,  as  seeim 
oeople  as  havinn  much  the  same  aradino  no  matter  what  was 
being  graded. 

When  PROBE  report  aradinns  were  compared  with  sueeryisor 
ratines,  thp  agreements  are  in  line  with  those  nenerallv 
reported  in  the  literature  whenever  anv  test  per-fnrmance  is 
compared  with  supervisor  oradinos.  The  anreement  between 
supervisor  ratims  of  observed  b heavier  and/or  performance 
and  PRORE,  for  30  white  telephone  operator  trainees  (^tudv  P) 
is  as  shown  in  Tapie  A. 
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The  anreement  between  supervisor  ratines  of  observed 
behavior  and/or  performance  and  ppopf  for  4T  Menro  telephone 
operator  trainees  (Rtudv  P)  is  as  shown  in  Table  S. 


TflBLF  h 


H e n t a 1 

Co  n f i - 

Coppp  ra  - 

t a h - 

1 e a d e r 

Grasp 

dence 

t i vpnoss 

i 1 i t \/ 

shin 

Knowledae  of 

. n 1 

.OS 

n s 

n s 

. 1 0 

Prod  activity 

. 0 1 

. 'Rh 

n s 

n s 

. 1 0 

Pride  in  Work 

n s 

. on  1 

. 1 ^ 

n c; 

n s 

Learnina  Rate 

n s 

. 1 0 

n s 

n s 

. 1 0 

0 ve  r a 1 1 

. 1 0 

. 1 0 

n s 

n s 

n s 

132 


STUDY  D;  Perfornance  data  were  acnuired  from  R'iS  emDlovPo 
candidates  of  U.S. Steel  (Texas  Works).  Dn  each  subiect,  an 
early  model  of  PROPf  was  processed.  These  raw  and  computed  data 
were  forwarded  to  the  Department  of  statistics,  Texas  a t 
University.  On  the  same  subiects,  U.S. Steel  provided  ioh  and 
training  data;  U.S. St eel  assured  Texas  A K *’  University  that 
employee  candidates  were  neither  hired  nor  not-hireri  on  the 
basis  of  PROBF  reports.  The  two  sets  of  data  (PPOpp  and 
U.S. Steel)  were  kept  entirely  separa+e  until  thev  reached  Texas 
A K University  for  analysis.  U.S.  Steel  data  consisted  of 
several  ratinas  performed  hv  supervisors,  and  certain  data-of- 
record  on  hired/not  hire,  absenteeism,  nrievances  filed,  and 
etc  . 

A brief  summary  of  the  findinns  ok  the  repression 
analyses  of  the  data  follows  as  Table  f-  . 

T A p L F A 

SUBJECTS  PROBE  PREDICTS  U.S. STEEL  PATIfJos  nr. 
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SUBJECTS  PROBE 
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hinh,  poorer 

s u pe  t V i '■-or 

ratims  ot 

emotional  s 

t a h i 1 i f V . 

Poorer  OROPE  , 

otter  supervisor 

ratims  o* 

emo  tional  stahi  1 

1 + V 

Emotional  stahi 

1 i tv  . 

Authoritarian  1 

e a d e r s h i p . 

Times  channel  f)  r>  p ^ r tm  e n t s . 

Emotional  s+ah| 

1 i tv  . 

Unreal is+iral Iv 

‘-iris^  poor  clu'-- 

+ f:'  r 

p 1 acpmen  . 

Unreal  istical  Iv 

n i 0 h , more 

absentee  i cm 

Unreal  istical  Iv 

hinh,  poor  wor  h 

performance 

Unreal  istical  1 " 

hinh,  more  sa'' 

et  V 

violations. 

Unreal  istical  Iv 

hinh,  poor 

acceptance 

asr I nnments  . 

Emotional  stahi 

1 i tv  . 

Better  oROnr,  nore  acts  of 
phvsical  ancression. 
Acceptance  overtime. 


134 


SUP.J  ECTS 


POOPF 


PPPPICT?  lI.E.S.  RATimps  OP: 


Product  ion. 
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‘Cental  Trasp 

Product  ion  , 
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Cooperati veness 
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'Production, 

'leg  ro 

Com'^ef  i ti  veness 

Product i on  , 

Negro 

Pon-*-  i dence 
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expected  direction  unless 


SL'M'-'ARV:  Although  perforf^ance  data  for  RROBF  nav  be  acnuired 

bv  non-professionals,  bv  the  use  of  te I ecoor u n i ca t i on  nets, 
cor'pyter  ■'ardware  and  conputer  softwares,  professional  iudoment 
and  experience  are  used  in  evaiuatina  those  data. 

PROPE  is  independent  of  the  constraints  o^  professional 
availability  and  cost,  qeooraphv,  personal  bias  of  the  exaniner, 
and  RROPE  is  designed  to  conserve  tine. 

There  is  no  practical  I i o i + to  the  nurber  of  persons  who 
can  be  evaluated  bv  '^R'^PE  in  a fast,  reliable,  valid  and  cost- 
effective  wav,  and  ts;.;  evaluation  predicts  ability  and/or 
willingness  to  profit  fron  a training  situation.  It  is  felt 
RROBE  could  be  of  significant  usefulness  in  the  evaluation  of 
applicants  for  nilitarv  duty  and/or  for  the  evaluation  of  those 
on  '"ilitarv  service. 

Pv  use  of  gregression  analysis,  base-1 i ne  performance  data 
of  RROPE  can  be  compared  with  anv  subsenuent  retest  performance 
data,  and  the  probabilities  of  change  can  be  calculated.  RROPE 
""av  also  be  used  in  serial  studies  to  determine  reversibility 
•'f  pathologic  changes  associated  with  aging,  disease  or  trauma. 
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I Qft7  Vinson,  P.P.;  A study  of  the  effect  of 
time  on  mental  efficiency  and  emotional 
stahilit','.  Am.  0(;y(~h(-,l.  Assn.  Annual 
v>eetinc,  Diy.  29,  Wash  i no  ton  PC 

|f>6f’  Vinson,  P.B.;  i^erformance  testine  as  i + 

relates  to  iniury.  WatN.  'Research  '^ouncil, 
Diyision  on  'fedical  'Sciences 

|o70  Vinson,  P.l^.:  Computer  eyaluation 

relatiye  to  iniurles.  12th  flat'l.  Conf., 
edical  Aspects  of  'Snorts,  Poston 

1971  Vinson,  D.P.,  Constant,  P.A.,  Oy ac he n b u s h , 
J.P.  & “olloy,  J.P.:  Automated  mental 

status  examination:  pre  and  post  druo 

effects.  Drua  Information  Assn.,  Computer 
Workshop,  Chicaao 

|Q72  Vinson,  P.B.:  Automated  assessment  of 

chances  in  human  performance.  Brand 
f’ounds.  Pay  lor  I'nlyersity  Colleae  of 
■’edicine,  January  21. 

|072  Vinson,  P.P.;  Telecommunications  and  the 
automated  mental  status  examination. 
fJat'  I . Telecommunications  Conference, 
i'ous  ton  . 

|073  Vinson,  P.B.;  Ban  computer  pronrammina 

predict  no s t -o p er a t i y e return  to  function? 
^acuity,  "What's  Wpw  in  Orthopaedics," 

Am.  Acad.  Orthopaedic  Burceons,  ‘'an  Antonio 

(073  Vinson,  P.P.  ‘i  ‘far  per,  B.P.;  prj|"r.  g 
mu  I f 1 - I a mu  ane  automated  psychiatric 
screenim  +ool.  Clyil  ''yiation  "edical 
Assn.,  o y e d a I a i a r a , " e x 1 c o 
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ASSESSMENT  SYSTEMS,  INCORPORATES 
"PROPE" 

OCTOBER  12  1973 
ASI-123-456789 


MENTAL  GRASP:  3.1  THE  ABILITY  TC  LEARN  FROM  A SYMrCLIC  REPRESENTATION 

OF  A PROCESS  OR  PROCEDU'^E.  THE  UTILITY  TO  UNDERSTANO  HOW  A SUP-SVSTFM 
RELATES  TO  A ICTAL  SYSTEM.  THE  A^'ILITY  TO  THINK  OF  ALTERNATE  SOLUTIONS 
TO  PROBLEMS. 


SELF  CONFIDENCE:  2.4  THE  INDIVIDUAL’S  FEELING  OB  EXPECTATION  THAT  HE 

WILL  MASTER  A PROCEDURE  AND/OR  PERFORM  AS  SUCCESSFULLY  IN  FUTURE 
AS  HE  HB3  IN  THE  PAST. 


COOPE.baTIVENESS:  b.O  IDENTIFIES  WITH  4UTHOBITY.  TAKES  DIRECTION  FROM 

THOSE  IN  AUTHORITY.  IDENTIFIES  WITH  CORPORATE  GOALS.  SUPORDINATES 
PERSONAL  GOALS  TO  CORPORATE  GCAL'S. 


EMOTIONAL  STA^ilitk*  4 .c  CCNSIDEBR  THE  OUTCOME  ^ppppr  pr  ACTS.  IS 
A=LE  TO  CONTROL  TFMPrp  4-r  jc  TCLEBAjr  FRUSTRATION.  SFTs' LONG-TERm' 

RAiwpR  tban  shobt-term  goals  for  himself. 


LEADERSHIP:  0.9  jpc  ficjLjyv  yr  y 

AND  TO  ENLIST  THE  COCPEBATICN  OF  0 


KE  COMMAND,  TO  INSPIBp  CONFIDE^CE■ 
HS'’S  . 


IN  EACH  CATEGOBv  a '"atING  OF 

1 : SIGNIFICANTLY  ABOVE  AVERAGE 

^ : A^rvL  AVERAGE 

3 : WITHIN  NORMAL  LIMITS 

A : FELOW  AVERAGE 

5>  : Sn^lJ  fic^I^XLY  FELOW  AVEBAGE 


THIS  avalysIS  may  REFLE.CT  THE  SUBJECT'S  PHYSICAL  A\R  EMOTIONAL  STATUS 
AT  THE  TIME  OF  TESTING  AND  SICUl.R  NOT  FT  USEE  as  A SOLE  CB]JERICn  FOB 
ACTION  OR  DLCISICN.  THIS  REPCRI  WILL  SISNED  ONLY  WHEN  TEST 

findings,  SCC^ING  Apiy  calculations  on  which  PEPOBT  is  based  have  f-EEN 
VERIFIED.  - HKvc.  . rciv 


FIGURF  ? 


39 


Pinu^E  ^ 


ASSESSMENT  SVSTEMS,  INCOSPr°*TEr 
”PDX" 

OCTCTEP  IS  1973 
ASI  -1'’3  A56789 


LEVEL  OF  COGNITIVE  FUNCTION 

ATTENTION  IS  AT  THE  UPPE'*  LIMIf  OF  NORMAL  LIMITS. 
conceptual  THINKING:  A“I'_irv  70  ISOLATE  A COMMON  CH  AP  AC  TER  I ST  1 C 

AND  TO  SHIFT  F9CM  ONE  POINGIPLE  OF  CLASSIFICATION  TO  ANOTHER 
IS  AT  THE  LCWcS  LIMIT  OF  N^pmA-L  LIMITS. 

LEVEL  OF  PEPSONALITV  FLNCTIOf 

S"LF  CONFI OENC^":  LEVEL  OF  aspiRahen  is  WITHIN  NORMAL  LIMITS. 

EMOTIONAL  STAEILITY:  THE  PATIENT  NEIIITS  EMPLOYS  THE  MECHANISM 

OF  DENIAL  NOP  ACTS  CUT  AGGO^E'IVE  IMPULSES. 

EGO  FI  ''CTION:  PATIENT  'S  a:ilITY  and/C'"  WILLINGNESS  TO  IDErTIFY 

VIE''  THE  PREVAILIN.  C ..LTL'E-'' V^LI'p  Svejc^',  is  AT  THE  LO'aZR 
LIMIT  OF  NCSMAL  LIMITS. 

Acc::^r'MCE  of  TSEAT'''£:;r  plan 

PATIENT’S  accEPTANCl  OF  THE  0 00  Tf'^-PAli ;;  M pfi.aTICNSHIP  amp 
'WILLINGVESS  TO  COOPEPATs  WITH  It  E ST^RF  IS  LITHH  NORMAL  LIMITS. 

I'-PPESSION 

TO  EE  PULED  Ol'T: 

»i  r' »« T 

IF  FOPEGOING  NFSAUVE,  T!'E  OLlvlCAI..  ''lASNCSES  (IN  ACCORDANCt^ 

LIT)'  TON-ii)  APjT  p-^SSICTL'  TO 

Ajc'UITMEfT  ■’rACIION  OF  LIFE 

N ME  N T A L r I 0 0 P ■'  E 

'It't  ■'h'y’'icm.  an'-,  emotional  status 

^ ^ Or  T’.STI'O.  A“’_  S'-'^ULf  N^  T 'E  pE.G  as  c ‘'OL'^  CpIT-p^ON  FOR 

AOriO'  fp  ''F.C  I S 1 0“' . T'-'I'  PEPA-T  '.'ILL  ~ OM.v  '//ppN  TF^l  FINpINGP 

IN  1 AM  , CALCUL-'- I I p*'S  ' .'HIC*-  Ppp'PT  I"  SLO  HAVr.  ~EEN  VFPIFiro, 


■)  . 


Prospective  Chief  Petty  Officer 
Leadership/Management  Seminar 

Commander  G.  C.  Hinson 
United  States  Coast  Guard  Training  Center 


INTRODUCTION 


The  need  for  leadership  in  the  military  services  is  axiomatic.  At  all  levels, 
managers--those  with  a responsibility  to  get  a job  done--must  be  able  to  direct 
other  men  to  want  to  do  a job  well.  Recently,  within  the  Coast  Guard,  there 
has  been  increased  emphasis  placed  on  the  leadership  abilities  required  of 
E-7  through  E-9  Chief  Petty  Officers.  Late  in  1972,  the  Commandant  tasked 
the  Training  Center  at  Governors  Island,  N.  Y.  with  defining  the  elements  of 
leadership  a Chief  Petty  Officer  must  possess,  and  preparing  a "nuts  and 
bolts"  course  to  teach  these  prospective  leaders  this  managerial  skill.  This 
paper  outlines  the  procedures  that  were  used  in  development  of  such  a course. 

BACKGROUND 


What  is  the  essence  of  leadership?  Can  it  be  taught  in  a classroom?  Can 
this  be  taught  at  all?  Can  it  be  taught  to  everyone?  Can  men  be  transformed 
into  leaders?  A Questionnaire  sent  to  E-9  Chief  Petty  Officers  revealed  a 
leader  to  be  intelligent,  a specialist  in  his  rate,  imaginative,  dependable, 
capable  of  making  quick  decisions,  especially  under  pressure,  and  addition- 
ally one  who  inspires  his  men.  Could  we  develop  a course  to  produce  such  a 
person?  Is  this  the  man  who  could  get  the  job  done?  Research  in  this  field 
showed  that  many  leadership  courses  are  available.  Most  courses  are  designed 
to  shape  men  into  what  someone  thinks  leaders  ought  to  be.  In  general  many 
efforts  have  been  disappointing  and  a waste  of  time  and  money.  Even  extensive 
training  at  military  academies  has  failed  on  many  occasions  to  transform 
cadets  into  effective  leaders.  The  reasons  for  our  apparent  failures  should 
be  explored,  are  our  methods  of  instruction,  the  students  or  a combination  of 
these  two  factors  to  blame;  or  is  something  else  at  fault?  What  should  be 
changed  to  improve  leadership  training?  Many  studies  show  that  the  most 
important  dimension  of  a leader  is  his  relationship  to  his  subordinates.  He 
will  be  most  effective  when  his  men  respect  and  trust  him.  But  respect  and 
trust  are  built  upon  something  deeper  than  personality  and  friendship.  This 
made  a lot  of  sense  when  we  re-read  the  characteristics  cited  on  the  CPO 
questionnaire:  intelligent,  specialist  in  rate,  imaginative,  dependable, 

capable  of  making  a quick  decision,  especially  under  pressure.  It  is  clear 
that  respect  and  trust  must  be  resultants  of  performance  oriented  character- 
istics. Fred  E.  Fiedler  has  conducted  extensive  research  on  leadership  at 
the  University  of  Illinois.  His  studies  show  that  it  makes  little  sense  to 
speak  of  a good  or  poor  leader.  There  are  only  leaders  who  perform  well  in 
one  situation  but  not  in  another.  In  these  situations,  he  differentiates 
between  task-motivated  and  relationship-motivated  leaders.  Those  primarily 
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interested  in  building  personal  relationships  with  their  fellow  workers  are 
said  to  be  relationship  oriented;  those  interested  primarily  in  getting  the 
job  done  are  considered  task  motivated.  Fiedler  also  asserts  that  a task- 
motivated  leader  will  have  a greater  probability  of  success  than  a relation- 
ship motivated  leader.  Based  upon  our  analysis  of  the  literature  available, 
a course  in  leadership  should  stress  (1)  the  psychology  of  motivation  and 
the  satisfaction  of  doing  a job  well,  (2)  analysis  and  control  of  leadership 
situations,  and  (3)  building  a relationship  of  mutual  respect  between  the 
leader  and  members  of  the  group.  This  was  the  basis  for  the  Coast  Guard 
Leadership  Course  and  our  challenge  was  to  develop  realistic  rather  than 
idealistic  leaders. 

HISTORY  OF  DEVELOPING  THE  COURSE 


E-7  through  E-9  Chief  Petty  Officers  are  Coast  Guard  front-line  managers. 

They  most  frequently  fill  middle  management  roles  serving  to  clear  the  way 
for  command  actions  and  letting  the  command  know  how  the  crew  feels.  But 
on  the  other  hand.  Chiefs  are  assigned  smaller  units  as  Offi cer-in-Charge 
and  as  Executive  Officer.  These  possible  assignments  require  having  both 
a narrow  specialized  knowledge  in  a technical  field  and  broad  knowledge  of 
the  service  in  general.  They  also  must  be  able  to  communicate  vertically 
and  horizontally  within  the  conmand.  Our  leadership  course  was  tailored  then 
to  meet  these  needs.  We  sought  also  to  provide  prospective  Chiefs  with  a 
basis  for  decision  making.  A leader  has  three  methods  of  exercising  his  skill 
he  can  be  authoritarian,  democratic  or  depend  upon  his  men  to  exercise  re- 
sponsibility and  good  judgment  to  get  things  done.  Although  the  good  leader 
must  use  all  three  techniques  to  effectively  operate,  current  literature 
promotes  democratic  leadership  or  group  dynamics  as  the  preferable  choice. 

This  conflict  results  in  confusion.  To  quote  the  Harvard  Business  Review: 

"often  he  [the  leader]  is  not  quite  sure  how  to  behave;  there  are 
times  when  he  is  torn  between  exerting  'strong'  leadership  and 
'permissive'  leadership.  Sometimes  new  knowledge  pushes  him  in 
one  direction  ("I  should  really  get  the  group  to  help  make  this 
decision"),  but  at  the  same  time  his  experience  pushes  him  in  an- 
other direction  ("I  really  understand  the  problem  better  than  the 
group  and  therefore  I should  make  the  decision").  He  is  not  sure 
when  a group  decision  is  really  appropriate  or  when  holding  a staff 
meeting  serves  merely  as  a device  for  avoiding  iiis  own  decision 
making  responsibility." 

We  felt  a Chief's  decision  making  ability  would  be  proportional  to  the  in- 
formation he  had  to  draw  upon.  Consequently,  we  stress  the  forces  operating 
in  the  decision  making  process. 

(1)  fiscal  restraints 

(2)  responsibility  and  authority 

(3)  legal  boundries 
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(4)  military  considerations 

(a)  chain  of  command 

(b)  personnel  evaluation 
(r'  discipline 

The  successful  leader  is  aware  of  all  relevant  forces  and  can  put  them  into 
perspective  when  making  a decision.  He  is  flexible  enough  to  not  fear  change, 
when  change  may  be  indicated.  Therefore,  the  Coast  Guard  Leadership  Course 
emphasizes  analysing  situations,  knowing  available  options,  and  selection 
of  efficient  strategy.  Our  course: 

1.  Supplies  information  (or  indicates  where  information  can  be  found) 
concerning  relevant  forces. 

2.  Teaches  analysis  techniques  to  enable  one  to  know  self,  subordinates, 
superiors  , and  peers . 

3.  Stresses  the  selection  of  the  most  appropriate  and  productive  leader- 
ship techniques , and 

4.  Provides  the  opportunity  to  develop  communication  skills. 

SUMMARY 


Our  goal  at  the  Training  Center  is  to  produce  a realistic  leader;  one  who 
is  well  grounded  in  Coast  Guard  programs,  procedures,  policies,  and  problems; 
one  who  knows  his  importance  in  the  management  of  resources;  one  who  is 
sensitive  to  the  needs  of  his  subordinates  and  his  superiors.  We  feel  that 
our  four  week  Leadership  Course  can  achieve  these  goals. 


APPENDIX 


Outline  of  Curriculum 
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Enclosure  (1)  to  CO  TRACEN  Ltr  1510  dtd 

PROPOSED  PROSPECTIVE  CHIEF  PETTY  OFFICER  LEADERSHIP/MANAGEMENT  SEMINAR 


FIRST  WEEK 
Monday 

a.  Introduction  to  the  training  center  and  check  in. 

b.  Greeting  by  Commanding  Officer,  Execuu^ve  Officer,  Senior  Chaplain 
and  the  President  of  Governors  Island  Chief  Petty  Officers  Association. 

c.  Explanation  of  the  seminar's  methods  and  goals  - emphasizing  skill, 
not  theory. 

d.  Administer  the  Leadership  Opinion  Quectionaire 

e.  Present  historical  leadership  studies. 

Tuesday 

a.  In  depth  course  orientation;  How  to  accomplish  tlie  job 

b.  The  psychology  of  motivation  including  perception,  role  identification, 
communications  and  role  playing. 

c.  Achievement  motivation  including  security  vs.  challenge  quiz  and 
review  (quizzes  are  personality  preference  measures) . 

Wednesday 

a.  Continuation  of  achievement  motivation  including  Machiavellianism 
and  summary. 

b.  Leadership  styles^  Fiedler  Development. 

c.  Leadership  Matrix  and  review  Leadership  Opinion  Questionaire  questions. 


Thursday 

a.  Situation  evaluation  and  solution  choosing  - obtaining  facts  prior 
to  decision  making. 

b.  Return  LOQ  and  discuss  the  results. 

c.  Leadership  games,  Including  "Surviva]  on  the  Moon". 

Friday 

a.  The  importance  of  communicating,  encouraging  suggestions. 

b.  Retake  and  review  of  LOQ. 

c.  Review  and  di-':;uss  pain  research. 

SECOND  WEEK 


Monday 

a.  The  Coast  Guard,  including  the  movie  "The  Eighth  Mission",  Coast 
Guard  Organization  and  the  Chain  of  Command. 

b.  Career  Counselling  which  includes  methods  of  counselling,  self 
understanding,  human  understanding,  evaluation  of  people,  basic  sales- 
manship^ effective  listening,  counselling  problems,  interviewing  and 
specific  areas  of  CG  personnel  policies  (rotation,  etc),  V.A.  assistance, 
social  security,  insurance  survivors  benefits  and  annuities,  personal 
financial  management,  civilian  careers  both  after  retirement  and  the 
current  job  market. 
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Tuesday 

a.  Drug  abuse  Includlag  types  of  drugs  In  use  and  the  reasons 
people  turn  to  drugs. 

b.  Continuation  of  career  counselling. 


Wednesday 

a.  Drug  abuse  continued;  alcohol,  and  the  Drug  Exemption  Prograa. 

b.  Discipline  and  grievances. 

c.  The  Uniform  Code  of  Military  Justice. 

Thursday 

a . Human  Relations 

b . Body  Language 

c.  Fundamentals  of  recruiting 


Friday 

a.  Human  Relations 

b.  Public  speaking 

c.  Public  information 


THIRD  WEEK 
Monday 

a.  Human  relations 

b.  Budget  process  Including  Sub-Head  30,  planned  projects  and  procurement, 
cost  consciousness 

c.  Allowance  lists 

d.  Procurement,  Including  advance  planning,  work  lists  vs.  procurement 
lead  time. 

Tuesday 

a.  Human  relations 

b.  The  Personnel  Manual  (CG-207) 


Wednesday 

a.  Human  relations 

b.  Title  "B"  property 

c.  Personnel  records 

d.  Health  records 


Thursday 

a.  Pay  and  Allowances  - pay  office  proceedures 

b.  Travel  - household  effects 

c.  Leave,  leave  rations,  computation,  types 

d.  CHAMPUS  benefits,  elegibility,  orthodontics 

e.  Medical 

f.  Retirement 
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Prlday  ^ 

a.  Educational  opportunities,  Including  sevlce  schools,  CGI,  USAFI 
and  off  duty  education. 

b.  Advancements,  including  career  patterns,  servicewide  exams.' 

c.  uniform  regulations. 

d.  training  procedures. 

FOURTH  WEEK 
Monday 

a.  Organization,  delegation  and  planning  of  relatively  complex 
tasks.  Including  long  range  goals  and  intermediate  objectives  and 
the  need  for  follow  up. 

b.  Clerical,  including  effective  writing,  correspondence  manual. 
Tuesday 

a.  Forms,  reports  and  publications. 

b.  Classified  material,  handling  stowage,  access  and  the  need  to  know. 


Wednesday 

a.  Records  keeping 

b.  Rules  of  leadership. 


Thursday 

a.  CPO  responsibility  up  and  down  the  line:  a general  summation 

of  what  a CPO  is,  ranging  from  accident  prevention  through  experience 
to  rumor  control  by  not  participating  in  speculation;  a chief's 
presence  in  the  work  area  not  the  CPO  mess;  supporting  the  service 
and  superiors  and  not  adding  to  gripe  sessions  with  junior  enllsteds. 

Friday 

a.  check  out 

b.  Graduation  - CO  or  XO  address  class  - theme  "Personal  Integrity 
and  Ethics". 

ADDITIONAL  TRAINING  AS  DESIRED  (rJTGHTS) 

1.  Self  Taught  Reading  Improvement 

2.  Self  Taught  Mathematics  Improvement 

3.  Self  Taught  Spelling  Improvement 

TENTATIVE  SEMINAR  TEXTS 

1.  "What  Every  Supervisor  Should  Know",  Lester  R.  Bittle,  Second 
Edition,  1969,  McGraw-Hill  Inc. 

2.  Naval  Leadership,  US  Naval  Institute 

3.  "Right  Down  the  Line",  Charles  A.  Pearce,  Ed,  1955,  Arrow  Books,  Inc. 

4.  Various  articles  on  motivation,  achievement  and  leadership. 
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?:?/\I!SC^iIDSirTAL  M^DITATIOIJ  (TO): 
A TIIR’^AT  OH  THCKTIIOUH 
Ii:  FUTUHS  FHHS''r;!:HL  HVALUATK'N 


H.  0.  V/aldkoetter 

Cor.sultant,  Educational  Rr  Personnel  Systems 
Indianapolis,  Indiana 

'tow  Can  Personnel  Evaluation  For  Humctn  Hesourcee  3e  Humanly  Oriented? 

The  wide  expanse  that  is  perceived  when  we  begin  to  think  of  ways  to 
promote  the  best  uses  of  human  resources  includes  so  much  for  attention,  we 
may  be  disturbed  by  the  comrlexity  of  this  challenge.  The  human  subject, 
worker  or  friend  is  not  easily  directed  in  the  most  predictable  channels 
without  occasionally  interfering  with  our  programed  goals  by  time,  territo- 
rial or  value  differences.  In  behavioral  modification,  training,  and 
other  conditioning  experiences,  a rath  is  sought  which  can  give  desirable 
ethical  and  nroductivc  results  v;ith  ut  the  majority  of  persons  involved 
resisting  the  direction  in  which  managed  or  compelled  to  follOT/. 

ver  the  years  in  the  endeavors  addressed  under  the  quest  for  adequate 
personnel  evaluation  techniques,  there  has  been  much  concern  about  evalua- 
tion v/hich  can  reasonably  assure  that  a person  with  acceptable  skill  is 
selected  for  ■••hatever  epportunites  found  available.  This  goal  has  not 
diminished  as  greater  and  greater  effort  has  been  made  to  get  at  skill, 
ability,  interest  and  personality  factors  which  are  believed  to  be  funda- 
mental sources  of  variance  affecting  behavior.  As  a position  guiding  mea- 
surement design,  a creeping  doubt  is  frequently  dismissed  whenever  the 
problem  of  motivation  is  nosed  or  the  "sense  of  timing"  awareness  shown, 
which  both  seem  to  influence  hov/  skilled  behavior  is  brought  to  bear  in  any 
nroblem  situation. 

/ith  some  assumntions  about  what  have  been  generating  forces  for 
nersonne]  evaluation  — measuring  individuals  primarily  with  paper-pencil 
tests  in  achieving  some  type  of  criterion  behavior  — there  remains  also 
a principal  question  regarding  the  desire  and  persistence  needed  to  main- 
tain behavior  after  the  record  of  achievement  and  aptitude  is  indexed, 
■/hether  wo  view  persons  measured,  getting  ready  to  be  measured,  or  at  a 
specified  interval  after  measurement,  a constraint  is  still  present  that 
makes  one  feel  uneasy  to  the  degree  that  a strong  level  of  continuing 
motivation  is  not  foreseeable  unless  very  high  qualifying  standards  8U*e 
someh'w  apnlicd.  .'hat  choice  is  there  for  preparing  indenendent  behavior 
that  w^lll  directly  contribute  to  greater  efficiency  and  eventual  produc- 
tlvit’’  without  relying  upon  passive  and  coercive  methods  and  authcrltdarljan 
c ommand? 


If  you  will,  let  us  sunposo  there  are  techniques  by  which  we  can 
loighten  performance  by  iainrcvod  attention,  integrated  responses,  and  less 


Tjarticularly  eco-centBred  goals.  Under  favorable  conditions  Individuals 
V7ill  expend  their  intellectual  and  emotional  resources  to  attain  specif- 
ied goals  without  being  moved  by  harsh  instructions  and  built  in  punish- 
ment should  they  not  comply  to  set  schedules.  Again,  if  you  will' suspend 
critical  analysis  for  a moment,  I ask  that  you  give  your  attention  to  the 
concept  that  the  most  convincing  teaching  and  experience  results  from  your 
having  permitted  yourself  to  be  so  engrossed  in  a practice  task,  your  sense 
of  awareness  Is  almost  wholly  fouided  by  the  culmination  of  the  action 
being  performed.  Y .'u  sock  only  to  determine  the  degree  of  execution  by 
which  you  might  achieve  a finer  shade  of  nor formance  and  in  some  'way 
allow  yourself  to  evaluate  the  results  in  terms  of  some  other  instructive 
experience.  Vhen  the  activity  d 'os  not  satisfy  you  v/ith  the  anticipated 
results  som.e  corrective  feedback  is  usually  anpliod  to  obtain  adjusted 
future  responses  t 'v/ard  the  goal  selected.  Geldom  did  critical  insight 
occur  where  one  could  ascertain  that  rav/  energy  was  not  the  primary 
motivating  force,  but  rather  with  the  proper  amount  of  energy,  coordinated 
and  well-intergratod  behavior  most  often  furnished  a wise  problem.-sol ving 
approach. 


Yan  an  Alternative  lie:  human  Awareness  to  Support  Measurement f 

You  must  rocor-nize  intr -'duct  r;.'  portion  of  this  paper  is  sotting 
the  stage  for  a statom.nnt  f my  purpose  and  tentative  problem  resoiutlin. 
>7hich  Is  t-  roouost  that  u ro'^’oet  on  better  means  by  v;hich  personnel 
evaluation,  c inica'-  r .'bjectiva  y-orrlcnted,  can  sh '.w  some  humanistic 
em.phasls  on  an  Indivilual's  state  -f  awareness  to  identify  the  individual's 
locus  of  psych 'logica.  contr-''.,  bef  ro  respc)nding  to  outer-directed 
commands  to  reach  giver,  task  standards  of  performance.  ne  way  to  encourafp 
Intrinsic  and  humanistic  motivation  to  perform  up  to  the  Individual's 
potential  is  to  maximize  fu.ncti-ning  of  the  overall  neurological  system. 
Although  intensive  classic  c.-nditi oning  and  hypnosis  or  autosuggestion 
are  potentially  within  the  sphere  of  utilization,  I do  v/ith  all  candor 
subr.it  that  the  ncr-o'yalizel  locus  of  control  fts  advanced  by  the 
technique  of  Transconlentai  Yedit-otion,  and  its  possible  variants, 
offers  the  type  of  pr  cess  through  •.vhich  learning  and  training  can  be 
selectively  acco'eratod  and  nurtured. 


■let's  T.ook  at  a '^cchnique  to  .itrongthem  Ferformanco . 

Transcendental  ’^editati -n  (71)  is  presented  to  y.;u  for  a futuristic 
look,  and  is  nr.obab'  ’ only  one  of  several  avenues  of  treatment  t'">  be 
utilized  to  ir-nrove  tno  sorer, i;y  f consciousness  so  that  wo  ma;.'  perform 
more  efficient':;-.  It  is  defined  as  a process  v/horein  the  individual 
sits  usually  in  an  upright  n sitiin,  as  comfJrtably  as  -possible,  eyes 
closed,  cffortlcssl,-/  'ets  the  mind  drift  toward  a mantra  that  v/as  assigned 
b;-  a teacl'.or  and  initiator  {Pisko,  197P)*  I would  not  tr;,-  to  even  give 
a c .Jmprehonsive  definition  of  mind,  other  than  to  sa--  it  comprises  that 
ccnscious  awareness  .jf  the  manifold  fields  'f  activities  being  mediated  by 
the  functioning  f the  brain  and  nervous  s;-3tem.  The  mantra,  as  mentioned, 
is  a sound  which  sets  im  .0  mo  s ^rt  of  commanding  passv;ord  that  seems  to 


shift  the  level  of  consciousness  to  let  go  of  value-laden  thought. 

Although  not  quite  In  the  same  way  as  hypnosis  does,  TO  produces  with  the 
mantra  applied,  an  almost  Immediate  change  In  the  mental  and  physiological 
functioning  (Wallace,  1970). 

This  short  definition  Is  practically  all  that  need  be  stated  about 
the  Identity  of  TO  as  a technique.  The  agreement  on  Its  potential  effect 
In  personnel  evaluation  is  yet  to  be  acknowledged.  So  let  us  exercise 
the  collective  imagination  to  the  point  that  we  may  admit  there  are  many 
contingencies  which  will  become  evident  in  future  developments  In  personnel 
evaluation,  testing  and  personnel  system  analysis.  Riding  the  crest  of  the 
wave  with  statistical  applications  and  computerized  innovations,  testable 
bshavlor  has  been  mainly  explained  as  a dependent  variable  In  response  to 
tasks  set  before  the  human  subject.  Behavior  not  directly  perceptible 
through  the  nearly  always  observable  channels  of  psychomotor  evaluation  is 
routinely  dlsnissed  as  not  in  the  territorial  interest  of  the  practicing 
behavioral  scientist,  flew  models  of  human  behavior  should  be  proposed 
to  account  for  altered  states  of  consciousness  and  connected  tohavior. 

How  may  these  states  tend  to  produce  systematic  variations  in  behavior  when 
all  possible  characteristics  in  selected  individuals  appear  to  be  similar? 


What  Related  Ideas  and  Functions  of  TO  Can  Be  Suggested? 

To  a certain  extent  I believe  Me  Clelland  (1973)  was  speaking  of 
something  bordering  on  a change  in  a theory  of  testing  when  he  specified 
"tests  should  involve  operant  as  well  as  respondent  behavior,"  and  "tests 
should  sample  operant  thought  patterns  to  get  maximum  generallzabllity  to 
various  action  outcomes."  He  goes  on  to  say  that  "the  tester  of  the  future 
is  likely  to  get  farther  in  finding  generallzable  competencies  of  character- 
istics across  life  outcomes  if  he  starts  by  focusing  on  thought  patterns 
rather  than  by  trying  to  infer  what  thoughts  must  be  behind  the  clusters  of 
action  that  come  out  in  various  factors  in  the  traditional  trait  analysis." 
Yet  Me  Clelland  has  only  seen  fit  to  approach  the  edges  of  the  testing 
domain,  and  balahces  precariously  to  not  imply  further  that  we  might  well 
study  those  psychic  Vciriables  which  impact  in  the  levels  of  consciousness 
and  thus  affect  the  thought  patterns. 

To  get  behind  thought  patterns  for  testing  purposes  will  give  inter- 
ested scientists  no  end  of  quizzical  moments  when  they  attempt  to  leeirn  abou 
the  "conditions  of  knowing"  that  many  of  us  were  taught  to  be  forbidden 
territory.  Today  observing  human  energy  research  trends  in  biophysics 
and  such  interdisciplinary  fields,  if  the  psychophysicist  does  not  emerge 
to  work  with  the  innovative  experimental  clinlcan  the  future  of  human 
understanding  and  measui’ement  could  well  be  removed  from  the  hsmds  of 
conservative  theorists  and  practitioners.  How  could  this  ever  happen 
experienced  psychologists  will  say  attempting  to  think  of  flaws  in  the 
logic  of  this  paper.  Very  well,  then,  I will  hazard  several  predictive 
views  under  which,  the  TO  influence  of  defined  levels  of  consciousness 
might  give  differing  insights  in  relation  to  how  certain  varieties  of  mea- 
sured behavior  could  be  interpreted.  Then,  if  these  predictive  views  do 
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not  BuggeBt  to  you  feasibio  stud"  problens,  I am  nistakenly  following 
heretical  research  dcvej.orr.ents,  and  rersonnel  evaluation  should  bo 
declared  only  a field  for  refining  nroven  psychometric  nethodolog;,' . 

Viewing  the  7!'  technique  aa  a supportive  treatment,  ’’et  us  visualize 
a situation  where  the  measured  outrut  of  an  individual  has  become  erratic 
and  this  person  seems  to  be  chronically  fatigued.  UniosB  there  is  some 
physiological  da.r.age,  he  should  regain  any  loss  of  applied  Bkill  where 
normal  lose  is  not  expected  to  accompany  a heavy  degradation  of  slCLll.  3y 
using  7!' ,'Val lace  (197^)  has  v/ith  advanced  meditators,  acting  as  their  own 
controls,  showed  changes  ir.  '7C.  (more  frontal  alnha  and  theta  waves),  and 
decreased  oxygen  c 'nsurintior. , heart  rate,  cardiac  output,  and  blood  lactate, 
with  increased  skin  resistance,  and  certain  restful  changes  greater  than 
many  hours  f sleep.  The  acquisition  and  recall  of  the  individual's  lost 
skill  should  nroceed  m're  swiftly  as  the  refreshed  subject  or  incumbent 
voluntarily  experiences  T'  and  makes  an  into  grated  learning  reaction 
after  the  ns;.-ch onh;'Giologicai  adjustment  induced  by  TK.  ■ ne  small 
pilot  study  (Abrams,  1972)  has  shown  support  for  a direct  relationship 
between  TV  and  quicker  acquisition  and  higher  recall  performance  on  a 
paired-associate  learning  and  recall  task.  A smoother  response  to  achieve 
certain  ner f ormances  c'uld  well  be  anticipated  then  as  tlio  individual 
self-directs  his  ^wn  quiet  conservation  of  noroonal  mental,  physical, 
and  emotional  resources. 


•Another  anplicntion  of  y'  could  be  viewed  in  contrasting  this 
technique  with  preparation  and  conditioning  for  intensive  problem- 
solving tasks.  In  training  and  education  systems,  where  severe  experiences 
tax  mental  and  nhysica  endurance,  shifting  levels  of  consciousness 
might  furnish  that  added  increment  .if  renewed  vigor  and  need  for  achievement. 
Ho7/  often  is  frustratisn  brought  about  in  learning,  and  defended  as  the 
need  to  "overcome  resistance",  only  to  induce  Inhibiting  effects  which 
extin, guish  desired  learning.  Vhi.’e  test-taking  anxiety  has  been  significantly 
lowered  in  a study  (Doucette,  1972),  other  reports  have  indicated  decreases 
in  negative  persona : it./  characteristics  and  expansion  of  av/areness  described 
as  self-actualization  (Kanellakos  and  Pellln,  1972).  Although  highly 
conditioned  responses  to  given  tasks  are  required  in  critical  behavioral 
sequences,  the  ability  t ' adapt  t '•  varying  circumstances  and  threats  to 
composure  could  be  enhanced  b;.-  'T'  training.  Finding  how  one  goes  about 
encouraging  the  trainee  and  job  incumbent  to  accept  readily  the  potential 
of  such  a technique  : :r  restoring  and  recreating  vital  energy  does  remain 
a lesson  to  be  learned  in  interfacing  individual  -and  group  dynamics. 

As  a different  research  vic’vpoint  to  establish  what  uses  suggestibilit'-’ 
may  provide  in  learnin.':  and  su.ctaining  job  standards,  may  we  not  attempt 
t ) distinguish  the  states  of  c ;nsciouEness  described  by  h'.'pnosis  and  TM.  ,ind 
try  to  examine  an.’  probable  similarities  or  unique  features  when  these 
techniques  are  introduced  into  a learning  enviroment.  Repetition  and 
conditioning  practices  are  normally  justified  in  training  and  management, 
as  essentia''  to  sustain  task  achievement,  and  the  individual  is  disciplined 
to  infer  that  m-otivati  m’’  pressures  are  largely  unavoidable.  assigning 
carefully  regulated  and  luntary  treatments  with  hypnotic  learning 
sequences,  the  human  bolng  ma.-  adapt  to  situations  in  which  he  can  monitor 
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hlB  own  hypnotic  condition  and  complement  his  further  awareness  by  the 
practice  of  TW,  A heightened  sense  of  human  understanding  is  hypothetically 
in  reach  and  the  behavioral  scientist  would  be  ready  to  analyze  aspects 
of  consciousness  all  but  left  to  imagination  and  theological  speculation. 
This  exploration  of  inner  space,  as  with  outer  space,  will  not  appeal  to 
everyone  and  to  some  conjures  up  threats  to  sanity  and  human  integrity. 

To  motivate  selected  individuals  to  undergo  experiences  which  to  date  have 
generally  proven  enriching  should  add  excitement  to  learning  while 
maxlAizing  performance  and  creativity. 

The  person  who  risks  his  own  well  being  and  those  material  things 
acquired  in  the  fully  conscious  realm  of  "getting  and  doing"  will  likely 
reconsider  goals  and  values  when  exposed  to  different  levels  of  conscious- 
ness in  which  spontaneous  trial  and  error  behavior  are  minimized.  No 
tendency  to  coerce  learning  and  task  behavior  should  be  tolerated,  but  When 
selected  individuals  master  the  experience  of  altering  their  conscir 
levels  of  awareness,  they  will  contribute  to  greater  creativity  and  compo- 
sure in  their  living  and  working  environments.  Both  political  and  reli- 
gious movements  have  long  profited  from  having  dedicated  individuals  and  a 
structured  method  for  preparing  themselves  to  face  challenges  as  potential 
opportunities.  In  effect  to  know  what  conscious  variables  in  altered  states 
of  awareness  open  a subject  to  greater  insight  for  accelerating  learned 
behavior,  I believe,  will  reveal  human  resources  only  seldom  tapped  and 
hardly  ever  articulated  as  hopeful  attributes  for  personnel  evaluation. 

•Vill  you  accept  the  existence  of  such  a technique  as  IW  and  does  it 
threaten  or  serve  as  an  optimistic  note  in  searching  for  well-integrated 
measures  to  predict  the  further  exploration  of  promising  human  resources? 


Can  We  Afford  to  Deny  Consciousness  in  Personnel  Bvaluatlon? 

Whenever  a projected  discussion  of  something  like  "states  of  conscious- 
ness" is  put  into  written  words,  the  fear  that  much  of  what  is  said  will 
seem  beyond  rational  comprehension  begins  to  be  very  constraining.  On  the 
other  hand  the  scientific  endeavor  to  plow  a new  section  of  thought  is  even 
more  provoking.  Whether  my  discourse  on  will  pique  your  interest  or 
disdain,  really  should  not  natter.  The  basic  purpose  is  to  let  a newer 
theoretical  and  experimental  alternative  be  expressed  so  that  personnel 
evaluation  does  not  become  too  self  satisfied  with  fundamental  principles 
of  human  performance  and  the  origin  of  measurable  qualities. 

Surely,  operant  behavior  can  be  interpreted  as  resulting  from  the  type 
of  environment  the  individual  gains  through  sensation  and  genetic  inheri- 
tance. However,  that  environmental  awareness  is  able  to  be  altered  by 
variables  which  are  not  immediately  perceptible.  ’Everything  from  the  electro- 
magnetic fields  and  atmospheric  changes  to  psychic  interference  by  adverse 
energy  patterns  developed  between  individuals  seen  to  give  us  pause  to 
wonder  what  does  cause  behavior  we  try  to  evaluate.  In  a recent  publication 
Rogers  ( 1973)  made  an  enlightening  review  of  some  new  challenges  in  psy- 
chology and  pointedly  questioned  "is  this  the  only  reality."  You  need  not 


151 


agree  with  his  own  brand  of  clinical  methodology,  yet  I feel  great 
significance  must  be  put  In  his  serious  query  about  needing  to  risk 
the  Investigation  of  paranormal  phenomena. 

An  approach  as  seen  through  the  channel  of  TM  can  help  open  the 
dark  door  so  prohibited  by  conservative  Investigators  when  inquiring 
about  the  other  reality,  .'/here  human  beings  In  all  walks  of  life  have 
carelessly  sought  a finer  awareness  with  drugs  and  other  illusory  sub- 
stances, that  behavior  has  led  to  many  unfortunate  Incidents,  At  least 
two  Army  general  officers  have  advocated  voluntarll'-  Introduced  present- 
ation of  W for  Interested  personnel,  but  only  after  being  initiated  into 
TO  practice  themselves  and  experiencing  personally  favorable  results 
(Sastnan,  1973). 

lev.’  horizons  are  going  to  be  attained  as  uses  of  TM  and  other  states 
of  consciousness  are  delved  into  to  discover  the  sources  and  functioning 
of  our  thought  patterns.  If  we  encounter  new  experiences  that  are  self 
actualizing  for  us  and  lead  to  finer  harmon.v  v/ithln  and  beyond  our  own 
degrees  of  awareness,  we  can  be  more  confident  in  understanding  individual 
behavior  we  are  assessing  and  how  human  resources  should  be  properly  used 
in  diversified  organizational  climates.  There  will  be  a constant  urge  to 
study  the  complete  human  system  in  the  interaction  with  environment  and 
productive  results  being  obtained.  The  emphasis  on  methodology  in  evaluat- 
ion cannot  bo  ignored,  hov/ever  the  finest  analysis  procedures  are  misguid- 
ing if  we  are  not  evaluating  the  skills  or  traits  we  think  should  be  under 
study. 


In  searching  for  various  ways  to  measure  and  apply  our  findings 
about  human  behavior,  the  goal  to  conserve  and  develop  a full  range  of 
human  resources  must  not  be  limited  by  lack  of  newer  concepts  and  subse- 
quently methods  in  personnel  eva’'uation.  During  a recent  lecture  by  a well 
known  astronaut  (Mitchell,  1973),  ho  stated  that  he  was  convinced  of  the 
need  to  study  the  spectrum  of  consciousness  displayed  by  the  human  organism, 
and  has  "rganized  such  a grouo  for  this  purpose.  iVith  this  kind  of  force- 
ful inquiry  taking  place  in  numerous  international  study  centers,  we  must 
agree  such  techniques  as  T!'  should  bo  of  some  interest  in  our  professional 
frame-of-reference.  There  are  human  abilities  which  apparently  transcend 
the  traditional  explanation  of  hovi  s!d.lls  are  normally  acquired  and  used. 

V/e  should  see  what  effects  those  abilities  have  on  our  theories  and  evalu- 
ation practices  even  if  the  prospect  seems  unsettling.  In  the  hope  that 
my  discussion  of  Transcendonta"'  l^editntlon  has  not  alienated  your  own 
creative  urges,  I end  'with  this  meditative  remark  attributed  to  Mlnstein: 
"...  Imagination  is  more  important  than  knowledge." 
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"IMPLICATIONS  OF  CARREL  INSTRUCTION  ON  MILITARY  TESTING"  A PRENARRATED  SLIDE  PRESENTATION 
BY  DR  RONALD  W.  SPANGENBERG  OF  THE  MEDIA  DESIGN  & EVALUATION  BRANCH,  TECHNICAL  TRAINING 
DIVISION,  AIR  FORCE  HUMAN  RESOURCES  LABORATORY 


AFHRL  TECHNICAL  TRAINING  DIV. 
MEDIA  DESIGN  & EVALUATION  BRANCH| 


Focus  frame 
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This  presentation  is  entitled  Implications  of 
Carrel  Instruction  on  Military  Testing,  Ron 
Spangenberg  speaking.  For  proper  synchronization 
you  should  now  be  seeing  the  specially  designed 
focus  frame  of  the  Media  Design  & Evaluation  Branch 
of  the  Technical  Training  Division,  Air  Force 
Human  Resources  Laboratory. 


CARREL  INSTRUCTION 


Student  working  at  standard 
carrel 


2 


What  does  carrel  instruction  mean  to  you? 


Basic  Electrical  and  Electronics 
School,  U.S.  Naval  Training 
Center,  San  Diego,  California 


Students  using  workbooks  and 
programmed  texts  at  "dry"  carrel;. 
Some  test  equipment  on  shelf. 


3 


For  the  Navy  student  at  San  Diego  it  means  a 
orogranimed  instruction  booklet  with  some 
items  of  electronic  test  equipment. 


Individual  Learning  Center 
U.S.  Army  Infantry  School 
Ft  Benning,  Georgia 

Study  carrel  containing: 

35mm  slide  orojector,  3/4"  V-mat 
format  video  cassette  player, 
Super  -8  film  cartridge  device; 
12"  color  video  monitor,  synch. 


c 


For  the  planners  at  Ft  Benning,  carrel 
instruction  means  media. 
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HRL  Media  D/E  Lab  -2 


iCollege  Library 

Mt  San  Jacinto  College 

Gilman  Hot  Springs,  California 


Study  carrel  containing:  35mm 
filmstrip  projector  and  reel  to 
reel  tape  playback  unit. 


5 


UPT  Learning  Center 
Williams  AFB,  Arizona 


Two-man  study  carrel  designed 
to  represent  T37  cockpit.  Audio 
selection  directs  the  student 
through  procedural  lessons. 


CARREL  INSTRUCTION  - SPANGENBERG 

AUDIO-VISUAL  STORY  BOARD  AFHRL/TT 


The  carrels  at  Mt  San  Jacinto  College  have 
prenarrated  filmstrips. 


Undergraduate  pilot  trainees  at  Williams  work 
with  A/C  control  panels 


6 


Student  watching  Audi  scan  with 
Security  Police  Training 
Program 


while  Security  Police  at  Lackland  can  view 
audiscan  presentations  at  a table. 


MEDIA  DESIGN  & EVALUATION  LAB 
ftFHRL 


partoon  figure  with  media 
?qui pment 


In  the  Air  Force  Human  Resources  Lab  the  Media 
Design  and  Evaluation  Branch  of  the  Technical 
Training  Division  has  been  involved  in 
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HRL  Media  D/E  Lab  -2 
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Weapons  Mechanics  Course 
Compulsory  Remedial  Training 
Lowry  AFB,  Colorado 


Students  viewing  prenarrated 
slide  presentations  at  standard 
carrel s . 


We  have  set  up  carrels  in  a learning  center 
presenting  prenarrated  slides  most  frequently. 


We  have  designed  and  built  a performance  aiding 
carrel  in  which  an  inset  panel  in  the  center  of 
the  working  surface  can  be  replaced  with 
snecific  oerformance  boards. 


Suci)  as  this  board  designed  to  refresli  the 
students  in  making  continuity  checks  with  the 
PSM  6 Mul ti-meter. 
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Three  conclusions  we  have  reached  are: 

(1)  carrel  instructional  materials  must  be 
developed,  produced  and  validated  locally. 
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(2)  carrel  instruction  is  not  necessarily 
performed  in  isolation.  We  feel  students 
frequently  should  work  together  as  in  this 
interactive  carrel  which  is  designed  to  help 
two  students  work  together  - or  for  an 
instructor  to  help  a student. 


(3)  the  third  conclusion  is  that  a carrel  is 
not  a piece  of  library  furniture.  This  portable 
carrel  can  be  rolled  to  actual  pieces  of  equipment 
and  provide  hands-on  instruction.  To  us  a 
carrel  could  consist  simply  of  a set  of  earphones. 


15 


INTERACTIVE  CARREL 

Interactive  carrel 
persons  viewing  an 
presentation 

with  two 
Audiscan 

14 

The  Audio  Visual  Instructional 
Display  System  module  being 
demonstrated  by  a secretary 


AUTHORING 


Carrel  with  two  Carousels,  8mm 
projector  and  audio  playback 
unit.  Random  access  slide 
selector  is  shown. 


CARREL  INSTRUCTION 


Student  working  at  standard 
carrel . 
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You  can  conclude  carrel  instruction  may  be  many 
different  events.  However,  there  are  three  areas 
of  evaluation  which  will  receive  increased 
emphasis  due  to  the  use  of  carrel  instruction. 


157 


HRL  Media  D/E  Lab  -2 


CARREL  INSTRUCTION 
QUALITY  CONTROL 


Student  v/orking  at  standard 
carrel . 
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CARREL  INSTRUCTION 
QUALITY  CONTROL 
MODULE  VALIDATION 

Student  vjorking  at  standard 
carrel . 
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These  three  areas  are  the  quality  control  of 
student  performance 


the  validation  of  modules  of  mediated  instruction 


CARREL  INSTRUCTION 
QUALITY  CONTROL 
MODULE  VALIDATION 

LEARNING  EFFECTIVENESS 

Student  working  at  standard 
carrel . 


and  using  evaluation  techniques  to  increase 
learning  effectiveness.  One  of  the  first 
implications  for  testing  in  the  area  of  learning 
effectiveness 


will  be  an  increase  in  the  development  and  use  of 
diagnostic  tests  to  direct  the  learning  of  the 
student.  The  student  the  instructor  or  the 
program  will  use  diagnostic  tests  to  select 
modules  for  optimal  paths  of  learning. 

Frank  Graiiam,  an  instructor  in  the  Weapons  Mechani 
Course  at  Lowry,  developed  some  prenarrated  sli’ 
modules  for  compulsory  remedial  training.  As  i, 
of  each  module  he  included  slides  such  as  these. 
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(1)  Pylon  wing  stations  are 
numbered : 

(a)  1,  2,  3,  4 

(b)  2,  4,  6,  8 

(c)  1 , 2.  4,  8 

(d)  1 , 2,  3,  9 
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The  student  is  given  about  30  seconds  for  each 
test  frame. 
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(11)  During  ground  operation  ... 

(a)  Turning  off  hydraulic 
pressure 

(b)  Installing  a lockout 

bol  t 

(c)  Installing  a ground 
safety  pin 

(d)  Reversing  breech 
sleeves 
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(8)  The  carrying  hooks... 

(a)  By  gas  pressure 

(b)  By  hydraulic  pressure 

(c)  By  electrical  solenoid 

(d)  Pneumat’cal ly 


Then  after  he  has  completed  the  test  he  will 
tlien  clieck  the  answers. 
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(8)  The  carrying  hooks... 

(a)  By  gas  pressure 

(b)  By  hydraulic  pressure 

(c)  By  electrical  solenoid 

(d)  Pneumatically 
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(11)  During  ground  operations  .. 

(a)  Turning  off  iiydraulic 
pressure 

(b)  Installing  a lockout 

bolt 

(c)  Installing  a ground 
safety  pin 

(d)  Reversing  breecii 
sleeves 
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Instructor  aiding  student  on 
MHU  83/E  Program  at  standard 
carrel 
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PSM  6 Mul  ti -meter  vn'th  red 
lead  being  inserted 


The  sound  track  v;ould  say  the  answer  is  (c). 
During  ground  operations  the  accidental  release 
of  loaded  stores  is  prevented  by  installing  a 
ground  safety  pin. 


Following  each  self  test  the  student  is  invited 
to  ask  the  instructor  to  explain  any  confusion 
which  he  may  still  have. 


Mr.  Raiford,  anotiier  Lowry  instructor  with  whom 
we  have  worked,  developed  a refresher  module  for 
making  continuity  cliecks  with  the  PSM  6 Multi -meter. 
Using  a mediated  nrogram  and  a special  board  in  a 
performance  carrel,  the  student  was  walked  through 
a PSM  6 continuity  check. 
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The  student  was  shown  an  open  circuit  check 
which  he  then  performed. 


Student  performing  continuity 
|check  with  Make/Break  switch 
jopen 


Then  he  was  shown  a closed  circuit  check  which 
he  performed. 


Student  performing  continuity 
check  with  Hake/Break  switch 
closed 


After  both  open  circuit  and  closed  circuit 
examples  were  shov/n  using  wires,  the 


Finger  pointing  to  wires  on 
Continuity  Check  Insert  Board 


Student  performing  continuity 
check  on  wire  6 showing  PSM  6 
reading 


student  then  performed  additional  continuity 
checks.  The  results  of  proper  checks  were  shown 
him.  Successful  comoletion  of  the  program 
required  the  self-evaluation  designed  with  the 
program. 
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In  another  area  a program  was  developed  by 
Ron  Filinger  on  densitometric  measurement. 


26.  21  STEP  WEDGE  - 11/11 


Final  frame  in  Densitometric 
Measurement  Program.  Tlie 
student  will  measure  11  out  of 
11  steps  on  a 21  sten  wedge. 
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After  the  student  is  siiown  the  correct  procedure 


Sensi tometric  Strip  No.  0995 


he  is  given  a sensitometric  strip  on  which  he 
performs  densitometric  measurements. 


Sensitometric  Strip  laying  on 
Jensi tometric  Measurement  Answer 
Sheet 


The  student  records  the  various  readings  w/iich 
ar.e  tlien  checked  by  an  instructor.  We  feel  that 
evaluation  should  be  integrated  into  each  module. 
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Learning  module  validation  will  become  increasingly 
objective.  Modules  will  be  systematically  evaluated 
to  insure  optimal  learning.  The  area  of  module 
evaluation  will  probably  exercise  the  innovativeness 
and  creativeness  of  military  testers  to  the 
greatest  extent. 
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In  module  development  and  validation  we  have  been 
experimentally  placing  technology  in  tlie  hands  of 
experienced  instructors.  I will  show  you  two  of 
tlie  techniques  we  have  examined.  The  first 
technique  is  dynamic  storyboarding  using  1/2  inch 
video  tape. 


Instructor  focusing  1/2"  video 
camera.  MAU-12 

Bomb  Rack  Program  cues  written 
on  chalk  board  in  background 
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CARREL  INSTRUCTION 
QUALITY  CONTROL 
MODULE  VALIDATION 


Student  working  at  standard 
carrel 


SHOOT 


[Instructor  shooting  Sony  Rover 
1/2"  video  camera  in  developing 
iMAU  12  bomb  rack  module 
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An  instructor  shoots  the  desired  performance  of 
the  job  being  trained.  After  review  and  feedback 
from  other  instructors,  and  possible  revisions. 


t'HOW 


Student  viewing  12"  video 
.lonitor  in  standard  carrel  over 
■1AU  12  bomb  rack  module 


the  video  sequence  is  shown  to  a small  number 
of  students.  These  students  are 
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EVALUATE 


Instructor  evaluating  student 
performance  on  MAU  12  bomb  rack 
module 
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then  taken  to  the  equipment  and  their  performance 
evaluated.  Any  difficulties  or  confusion  on  the  [ 
of  the  student  would  necessitate  some  revision  of  uie 
storyboard.  When  the  module  is  informally  validated, 
the  media  format  for  student  learning  is  selected. 

We  do  not  recommend  the  use  of  1/2  video  tape  as  a 
primary  teaching  media  but  normally  would  convert 
this  dynamic  storyboard  to  a prenarrated  slide  or 
Super  8 presentation.  Revision  or  validation  is 
completed  before  any  production  costs  are  incurred. 
When  the  instructor  learns  to  develoo  modules  which 
teach  visually,  another  storyboard  technique  can  be 
used . 


SHOOT 


Instructor  develooing  HJl  module 
is  shooting  with  Insta-matic 
camera 


Using  an  insta-matic  or  35mm  camera  he  can  shoot 
selected  critical  views. 
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SEQUENCE 


Instructor  seouencing  slides 
on  multiplex  slide  sorter 
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Then  when  he  receives  the  developed  slides  from 
the  photo  lab  he  selects  and  sequences  a set 
of  slides  portraying  what  is  to  be  taught. 


BHOW 


Instructor  projecting  slide 
rom  MJ  1 Module  for  students 


He  will  then  present  the  slide  sequence  to 
students  - developing  also  the  narrative 
associated  with  each  visual. 
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Student  operating  switches 
in  LT38  front  cockpit 
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Evaluation  of  synthetic  job  performance  can  take 
many  forms, 


49 


For  example  at  Ft  Carson,  video  tape  is  used  to 
evaluate  the  performance  on  the  Vulcan  Weapons 
System. 


Quality  control  can  be  built  into  many  modules. 
The  instructor  is  able  to  make  a performance 
check  on  each  member  of  the  team  of  four  students 
who  are  learning  the  A/C  preparation  procedure 
for  loading  an  AIM  7D  guided  missile  in  this 
quadraphonic  sound  module. 
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Students  can  then  be  evaluated  and  the  storyboar 
val idated. 
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The  third  area  of  increased  emphasis  in  evaluation 
related  to  carrel  instruction  is  quality  control. 
Quality  control  of  criterion  performances  will  be 
increasingly  emphasized.  Student  sampling  and 
extensive  student  performance  tests  can  insure 
the  objectivity  and  validity  of  student  performance 
checks . 
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Synthetic  job  environments  can  be  readily 
created  for  comparative  purposes.  The  use  of 
synthetic  performance  tests,  such  as 


this  cockpit  procedures  check,  should  increase 
because  of  their  potential  economy. 
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SYNTHETIC  JOB  ENVIRONMENT 


^Student  sitting  in  front  cockpit 
of  LT38,  a photo  mockup  with 
some  live  switches  of  F4C 
ai rcraf t 
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COCKPIT  PROCEDURES 


Student  at  dummy  rear  cockpit 
panel , LT38 


CARREL  INSTRUCTION 
QUALITY  CONTROL 


Student  working  at  standard 
carrel 


EVALUATE 


Instructor  evaluating  student 
performance  on  MJ  1 Module 
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objectivity  of  all  tests  be  established  through 
student  performance  testing.  An  example  of 
v/hat  I mean  is  now  being  conducted  at  Lowry  AFB. 


We  are  asking  the  question  of  whether  a picture 
test  can  substitute  for  a performance  test  in 
certain  kinds  of  tasks.  Using  the  same  modules 
of  instruction  we  are  comparing  two  ways  of 
testing. 


Following  exposure  to  the  learning  module 
Sgt  Osborne  takes  the  student  to  the  actual 
equipment  and  asks  the  student  to  show  him  the 
selected  items.  The  other  way  of  testing  permits 
group  testing  using  pencil  and  paper. 
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The  student  is  shown  a picture  of  the  item  and 
asked  to  select  the  correct  name.  These  ways  of 
testing  are  being  assessed  as  to  whether  they 
can  be  considered  equivalent. 
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The  use  of  carrel  instruction  appears  as  an 
opportunity  for  military  testing 
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to  provide  better  quality  control  of  student 
performance. 
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to  systematically  validate  learning  modules 


CARREL  INSTRUCTION 
QUALITY  CONTROL 
TOULE  VALIDATION 

Student  working  at  standard 
carrel 


CARREL  INSTRUCTION 
QUALITY  CONTROL 

Student  working  at  standard 
carrel 


CARREL  INSTRUCTION 


Student  working  at  standard 
carrel 


GROUP  TEST 
-PENCIL  PAPER- 


Instructor  testing  students  on 
MJ  1 parts  identification 
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IQUALITY  CONTROL 


MODULE  VALIDATION 


LEARNING  EFFECTIVENESS 

Student  working  at  standard 
carrel 


AFHRL 

TECHNICAL  TRAINING  DIVISION 
MEDIA  DESIGN  & EVALUATION 
BRANCH 
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and  to  increase  learning  effectiveness. 


Thank  you 
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Blnpham.ton,  New  York  13902 


An  increaslnr  pronortlon  of  the  v/ork  of  military  person- 
nel involves  participation  in  man-machine  systems,  such  as 
aircraft,  tanks,  shirs,  m.issiles,  and  command  and  control 
system.s , that  are  amenable  to  real-tim.e,  man-in-the-loor  simu- 
lation. In  such,  simulaticn.  Illustrated  in  Flnure  1,  the  man 
is  located  in  a renlica  of  his  normal  work  station,  and  pro- 
vided with  the  same  stlm.ull  — visual,  aural,  proprioceptive  — 
that  comprise  inputs  to  him  durlnn  com.bat  or  other  operations. 
He  reacts  to  these  stim.ull  , reneratinm,  suitable  outputs  or 
responses  — control  r.ovem.ents,  sv;itch  actuations,  verbal  com- 
munications, lor  entries  — ; the  simulator,  which  generally 
incorporates  a dlrltal  comnuter,  receives  and  acts  upon  these 
responses  to  m.odify  the  stimuli  on  a real-tim.e  basis.  Some 
simulations  are  deslrned  for  crews,  teams,  or  units,  rather 
than  sinrle  individuals.  For  som.e  command  and  control  or  mis- 
sile control  sl.mulatlons , the  operational  v/ork  station  is 
used,  with  synthetic  rather  than  real-world  Inputs  belnp:  sup- 
plied. This  is  sometlm.es  referred  to  as  stimulation  of  the 
display  system  at  the  work  station. 

Positions  amenable  to  simulation,  although  reoresentlng 
a small  minority  of  arm.ed  forces  personnel,  contribute  dis- 
proportionately to  the  effectiveness  of  each  of  the  arm.ed 
services.  Because  of  the  crucial  nature  of  these  positions, 
substantial  efforts  are  invested  in  the  selection  and  training 
of  men  to  fill  them,  and  in  the  assessm.ent  and  verification  of 
their  com.bat  readiness.  Simulation  plavs  a larre  and  increas- 
ing role  in  trainin,-.  For  aircraft  pilots,  sim.ulator  training 
is  far  less  costly  than  training  in  the  aircraft;  for  decision 
makers  in  com.m.and  and  control  systems,  it  allov/s  practice  of 
combat  situations  not  otherwise  available  in  peacetime.  Slm.u- 
lators  are  also  used  for  such  civilian  positions  as  airline 
flight  crews,  nuclear  power  nlant  operators,  air  traffic  con- 
trollers, and  autom.oblle  drivers. 

Simulators  have  been  a locus  of  measurement  for  a variety 
of  purposes,  indicated  by  the  following  sampling,  which  is 
meant  to  be  illustrative  rather  than  com.prehenslve : 

1.  Se lection . AFHRL's  Personnel  Research  Division  has 
a program,  under  wav  to  rieterm.lne  the  utility  of  the  Singer 
GAT-1  simulator  for  selection  of  pilots  for  undergraduate 
pilot  training. 
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2.  Training:  Feedback.  All  current  training  simulators 
have  provisions  for  furnlshlnr  the  Instructor  with  data  on 
trainee  performance.  Some,  such  as  the  ASUPT,  a T-37  simula- 
tor to  be  used  by  HRL  at  Williams  AFB  for  trainlnp;  research, 
and  the  2B2^,  a simulator  of  the  UH-1  helicopter  used  for 
transition  and  Instrument  training  at  Ft.  Rucker,  furnish  such 
data  directly  to  the  trainee  In  the  cockpit. 

3 . Deter.mlnlnm  the  Ff ^ect  of  "nvlronmental  Variables 
on  Human  Performance.  Rxam.nles  here  include  the  effect  of 
spacecraf t-llke  confinement  upon  behavior,  productivity  with 
various  work-rest  cycles,  and  the  effect  of  drur;s  upon  driving 
and  piloting  behavior. 

^ . Optlmlzlnr  and  Validating  the  Desirn  of  Operational 
Systems . In  the  aircraft  field,  simulators  are  extensively 
used  to  check  vehicle  handling  qualities  and  instrumentation. 

5.  Development  of  Procedures  and  Tactics.  These  range 
from,  communication  procedures  in  CICs  to  air-to-air  combat 
tactics . 

Table  1 shows  some  of  the  advantages  of  simulation  for 
various  kinds  of  m.llltary  systems;  they  can  largely  be  sum- 
marized in  terms  of: 

1.  The  breadth  of  situations  that  can  be  practiced, 

2.  The  precise  control  of  conditions  of  practice, 

3.  The  easy  availability  of  a broad  variety  of  reliable, 
valid,  and  face-valid  measures  of  man-machine  system  per- 
formance, and 

The  reasonableness  of  the  costs  associated  with 

1-3  above. 

With  all  these  advantares,  v;hy  aren't  simulators  used 
even  more  as  loci  of  psychometric  effort?  Much  of  the  re- 
mainder of  this  paper  will  address  that  general  question, 
focusing  on  flight  simulators  --  the  kind  with  which  I am 
most  familiar  — although  most  of  my  remarks  will  be  applica- 
ble to  other  kinds  of  simulators  as  well.  As  I see  it,  there 
are  six  factors  m.aking  measurements  of  perform.ance  on  simu- 
lators different  from  those  to  which  psychom^etrlclans  are 
accustomed,  and  hence  Inhibiting  their  use  of  simulators  as 
loci  of  psychometric  effort: 

1.  Historical  Primacy  of  Paper-and-Pencll  Tests.  Most 
early  testing,  and  certainly  almost  all  successful  early 
testing,  used  paper-and-pencll  Instruments;  excursions  Into 
areas  such,  as  reaction  time  did  not  orove  rroductlve.  Paper- 
and-pcncli  testin,'  is  chc:  n,  and  th.e  (io:..alns  it  covers  are 
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gene? ally  those  In  which  the  tester  Is  comfortably  superior 
to  the  testee.  Thus  the  basic  methodology  of  personnel  testing 
and  measurement  has  developed  around  paper-and-pencll  instru- 
ments to  the  point  where  no  one  thinks  It  Incongruous  that 
"Tests  and  Measurements"  textbooks  devote  upwards  of  80%  of 
their  space  to  paper-and-pencll  tests. 

2.  Multiplicity  of  Measures.  When  a psychometrician 
finally  decides  to  gather  human  nerformance  data  on  a simu- 
lator, he  suffers  an  embarassment  of  riches  — scores  or  even 
hundreds  of  measures  are  available,  all  possessing  face 
validity.  Figure  2 shows  data  obtained  during  the  landing  of 
a 707  simulator;  these  Items  represent  a selection  from  a 
much  larger  number  of  measures  available. 

3.  Lack  of  Parallel  Items.  In  a landing,  there  Is  Just 
one  Indicated  airspeed  at  touchdown;  to  get  a parallel  Item, 
another  landing  must  be  conducted.  This  lack  of  parallel 
items  within  a given  maneuver  contrasts  with  the  easv  avail- 
ability of  parallel  Items  for  most  paper-and-pencll  test 
domains . 

Difficulty  In  Comblnlnr  Item  Scores.  With  paper- 
and-pencll  tests,  the  num.ber  of  correctly  answered  Items 
usually  serves  as  an  overall  or  summary  measure:  occasionally, 
differential  weirhtlnr  Is  emnloyed.  Only  rarely  Is  as  complex 
an  algorithm  as  successive  hurdles  used  to  derive  a summary 
measure  from  Item  scores. 

Much  more  complex  algorithms  are  required  to 
characterise  nroflclency  In  tasks  such  as  landing  a 707.  For 
that  landing  task,  Items  such  as  Indicated  airspeed,  sink 
rate,  distance  from  runwav  threshold,  distance  from  runway 
centerline,  and  aircraft  attitude  (nitch,  heading,  and  roll)  — 
all  measured  at  touchdown  --  cannot  be  evaluated  Independent 
of  each  other,  as  paper-and-nencl 1 test  Items  are  menerally 
evaluated,  but  must  be  com.blned,  usinm  a rather  complex,  yet 
to  be  developed  algorithm. 

5 • Inapplicability  of  Conventional  Item  Analysis 
Techniques . Item,  analysis  Is  most  widely  used  for  multiple- 
choice  Items.  Kach  m.ultlple  choice  item  generally  represents 
a separate  behavior  sample,  and  the  alternatives  are  discrete. 
When,  as  they  often  do,  measurements  on  a simulator  cover 
various  aspects  of  a single  behavior  sample,  rather  than  sep- 
arate behavior  samples,  and  use  continuous  scales,  rather  than 
discrete  alternatives , conventional  item  analysis  Is  inappli- 
cable. 


6 . Unavailability  for  Measurem.ent  of  Behaviors  Mot 
Directly  Reflected  In  Machine  Inputs.  In  the  simulator,  the 
products  of  hurnmn  ‘eh'’v1or,  a?  the”  affect  the  machine,  are 
am.enable  to  ::  c:i.;ure' .ent , :.ut  the  pro:;^:-  by  ’.■.hlch  these 
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Inputs  are  made  are  not.  For  example,  In  a fllp;ht  simulator, 
we  would  normally  have  no  way  of  knowing:  whether  the  pilot 
used  an  efficient  visual  scan  pattern  In  checking  Instrument 
readings.  The  same  ob.-l ectlve  oerformance  could  be  achieved 
by  two  pilots  v/lth  very  different  Instrument  scan  patterns  — 
one  using  an  efficient  pattern  and  thus  minimizing  his  work 
load,  the  other  com.pensating  for  a less  efficient  pattern  by 
working  harder.  When  dealing  with  a simulator  that  enables 
measurement  of  such  a large  number  of  aspects  of  vehicle  or 
system  performance  by  merely  specifying  the  parameters  to  be 
measured.  It  Is  all  too  easy  to  forget  about  those  aspects  of 
human  performance  not  directly  reflected  In  machine  inputs. 

By  shifting  our  emphasis  from  paper-and-pencil  testing 
to  the  kind  of  performance  tests  for  which  the  simulator  Is 
such  an  appronriate  vehicle,  we  would  go  far  toward  meeting 
the  objectives  of  critics  of  our  current  testing  posture, 
su^h  as  McClelland,  v;ho  tells  us  to  test  for  competence, 
rather  than  intelligence,  and  that  "the  best  testing  Is 
criterion  samnling"  (McClelland,  1973,  p.  7)  and  O'Leary,  who 
advocates  the  "The  advantages  of  job  slm.ulatlon  approaches  to 
selection  over  technlaues  lacking  content  validity"  (O'Leary, 
1973,  P.  1^9). 

In  conclusion,  we  have  a long  way  to  go,  In  exploiting 
simulators  as  mieasurement  tools,  before  their  one  basic 
weakness  — inability  to  m.easure  behavior  not  directly  re- 
flected In  machine  inputs  — becomes  a truly  limiting  factor. 
By  refusing  to  be  shackled  to  psychometric  practices  appro- 
priate only  to  paper-and-pencli  tests  , we  can  exploit  this 
powerful  tool  to  measure  a wide  range  of  Individual  and  team 
behavior,  obtaining  relevant,  reliable,  valid,  and  socially 
useful  data  at  modest  cost. 
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ILS  LANDING  MCI  RVY  36 

INITIAL  COND  REF  DATA 
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The  teletypewriter  printout  actually  took  the  form  of  a single 
long  column;  it  is  formatted  in  two  columns  to  save  space. 


Figure  i>  DATA  OBTAINED  DURING  LANDING  OF  A 707  SIMULATOR 
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ADVANTAGiS  OF  SIMUWTORS  OP  VARIOUS  KI!TOS  OP  MILITARY  SYST!>1S 
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THE  PERSONNEL  AND  TRAINING  EVALUATION  PROGRAM: 

A Working  Program  for  Improving  the  Efficiency  and 
Effectiveness  of  Fleet  Ballistic  Missile  Weapons 
System  Training 

Part  I Scope  and  Achievement 
by 

Lcdr  Clarence  L.  Walker  USN 

Central  Test  Site  for  the  Personnel 
and  Training  Evaluation  Program 

INTRODUCTION 

In  April  of  1969  the  Chief  of  Naval  Operations  issued  an  instruction 
which  brought  into  being  the  Fleet  Ballistic  Missile  Weapons  System 
Training  Program.  This  Program  was  aimed  at  improving  the  effective- 
ness and  efficiency  of  training  in  the  areas  of  the  missile,  missile 
launcher,  missile  fire  control  and  navigation  on  FBM  submarines  and 
at  their  support  activities.  Specific  gains  looked  for  were  the  short- 
ening of  the  replacement  pipeline,  provision  for  advanced,  vice 
refresher  training  at  the  ship's  homeports  and  standardization  of  the 
advanced  training  available.  A built-in  part  of  the  FBM  Weapons 
System  Training  Program  was  a monitor  called  the  Personnel  and  Train- 
ing Evaluation  Program  (PTEP) . The  program: 

"provides  the  organization,  procedures,  and  responsibilities 
required  to  accomplish  the  qualitative  assessment  of  personnel 
knowledge  and  skill  levels  for  officer  and  enlisted  personnel 
during  replacement  training  and  while  assigned  to  SSBNs,  FBM 
Tenders,  and  FEM  training  facilities.  The  PTEP  also  provides 
for  evaluation  of  training  facilities,  hardware,  documentation, 
and  courses  of  instruction  for  use  as  the  basis  for  implementing 
improvements  in  training  and  in  all  elements  of  the  FBM  Weapons 
Systen,  Training  Program." 
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Various  aspects  of  PTEP  have  been  presented  at  the  last  two  meetings 
of  this  association.  For  these  presentations  PTEP  was  in  the 
speculative  stage.  This  paper  addresses  the  early  working  stages. 

To  understand  PTEP,  however,  it  is  necessary  to  look  at  the  climate 
in  which  it  operates. 

PTEP  is  presently  limited  to  the  Poseidon  Missile  community,  which 
consists  of: 

1.  A replacement  training  site  - Naval  Guided  Missiles  School 

Dam  Neck,  Virginia 
Dam  Neck,  Virginia 

2.  Two  off-crew  training  sites  - Naval  Submarine  School, 

Groton,  CT. 

FBM  Submarine  Training  Center, 
Charleston,  SC. 

3.  Two  submarine  tenders 

4.  Twenty-six  FBM  Submarines  each  with  2 crews 

The  subm.arine  crews  com.prise  the  majority  of  the  personnel  and  absorb 
the  vast  majority  of  the  effort. 

FBM  submarin-’s  utilize  a two-crew  concept:  One  crew  mans  and 

operates  the  s jbmarine  while  the  other  crew  is  available  at  an  off- 
crew  site  for  training.  Planning  revolves  around  a six-month  patrol 
cycle.  This  consists  of  one  patrol/refit  period  and  one  training/ 
leave  period.  Each  man  remains  aboard  for  approximately  six  patrol 
cycles  or  three  years;  he  is  then  rotated  to  shore  duty,  frequently 
at  a training  site;  following  this  he  returns  to  sea  again.  All 
personnel  selected  for  FBM  Weapons  and  Navigation  Training  are  of 
above  average  intelligence.  The  submarine  personnel  have  an  initial 
six  year  obligation,  with  the  exception  of  the  Torpedomen  responsible 
for  the  launch  tubes.  First-term  reenlistments  run  in  excess  of  40%. 
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These  then  are  the  personnel  involved  in  the  FBM  Weapons  System  Train- 
ing Program.  The  training  program  itself  is  not  unique  in  its 
academic  concepts,  but  it  is  innovative  in  the  area  of  program 
management.  As  with  any  viable  training  system,  the  FBM  Weapons 
System  Training  Program  first  provides  training  objectives.  It  deter- 
mines where,  when  and  by  whom  the  learning  is  to  take  place;  it 
provides  the  learning  opportunity,  and  then  assesses  the  quality  of 
the  end  product. 

PERSONNEL  PERFOR>!ANCE  PROFILES  AND  TRAINING  PATH  SYSTEM  (PPP) 

The  Personnel  Performance  Profile  (PPP)  is  the  basic  building  block  of 
the  s-ystem.  The  profiles  are  for  the  most  part  developed  by  hardware 
contractors  and  state  in  a standardized  format  the  skills  and 
knowledge  necessary  to  operate  and  maintain  a system  or  equipment. 

A system  called  The  Training  Path  System  assigns  to  each  FBM  Navy 
Enlisted  Classification  (NEC)  the  profile  items  and  levels  of 
achievement  appropriate.  This  same  system,  by  use  of  profile  items 
and  levels  of  achievement,  assigns  material  to  replacement,  advanced 
and  on-board  training.  Curricula  are  prepared  using  the  profiles 
and  Training  Path  System.  Informal  training  materials  in  support  of 
the  profiles  have  been  developed,  and  watch  qualifications  require- 
ments have  been  keyed  to  the  profiles  and  levels  of  achievement.  The 
FBM  Weapons  System  Training  Program  provides  a system,  whereby,  the 
required  knowledge  for  an  individual  or  the  specified  content  of  a 
course  can  be  conveniently  identified  by  listing  profile  items  and 
achievement  levels. 
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PTEP  ADMINISTRATION 

The  Personnel  and  Training  Evaluation  Program  is  administered  by  35 
military  personnel  and  a contractor  civilian  staff  of  about  10  people. 
One  officer  and  an  enlisted  staff  of  experienced  technicians  are 
stationed  at  each  training  site  to  support  the  testing  and  evaluation 
tasks.  T.he  bulk  of  the  enlisted  staff  is  at  the  Central  Test  Site  in 
Dam  Neck,  Virginia.  The  program  has  a broad  charter  of  evaluation. 
Evaluation  requires  facts.  PTEP  gathers  its  facts  through  examina- 
tions, collection  of  personal  training  information,  ship  and  equipment 
operational  history,  and  review  of  curricula  with  their  supporting 
material.  Examinations  are  the  primary  source  of  data. 

Two  separate  types  of  examinations  are  administered  by  PTEP.  One 
measures  the  knowledge  of  an  individual  against  knowledge  required  of 
him  by  the  Training  Path  System.  This  is  called  a System  Achievement 
Test,  or  SAT.  The  other  typo  of  examination  measures  the  performance 
of  a course  against  its  Training  Path  System  requirements.  This  is 
called  a Course  Achievemen'  .est  or  CAT.  In  both  cases  examination 
content  is  based  on  PPP  items  and  achievement  levels  prescribed  by 
the  Training  Path  System. 

Course  Achievement  Tests 

Course  Achievement  Tests  are  designed  to  measure  both  the  performance 
of  an  individual  in  a course  and  the  performance  of  the  course  against 
its  prescribed  content.  The  instructor  does  not  see  the  test  until 
it  is  time  to  administer  it  and  so  has  no  opportunity  to  "teach"  the 
test.  Standardization  of  training  between  sites  can  also  be  measured 
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with  a Course  Achievement  Test.  Few  CAT's  have  been  produced  to  date 
primarily  because  the  standardized  courses  they  support  have  not  been 
in  place.  No  meaningful  analysis  has  been  possible  with  the  results. 

System  Achievement  Tests 

The  greatest  field  of  endeavor,  and  heart  of  the  Personnel  and  Train- 
ing Evaluation  System  from  the  testing  point  of  view,  is  System 
Achievement  Testing.  The  tests  were  implemented  incrementally  by  NEC 
over  a period  of  9 months,  starting  in  August  1971,  To  date  there  have 
been  more  than  4,000  administered.  Each  of  the  several  tests  in  use 
is  designed  to  cover  the  full  scope  of  the  knowledge  required  for  a 
specific  Navy  Enlisted  Classification.  Content  is  determined  using 
the  Personnel  Proficiency  Profiles,  and  the  Training  Path  System. 

The  tests  consist  of  two  parts,  a knowledge  part  consisting  of  from 
180  to  360  multiple  choice  questions  and  a skill  part  whih  uses  a 
paper  work  fault  isolation  technique  which  has  been  developed  by  the 
Data-Design  Laboratories,  Depending  on  the  NEC  involved  the  tests 
take  3 or  4 three  hour  testing  sessions.  Each  of  the  testing  sites 
is  connected  by  a data  link  with  a central  computer  complex.  The 
tests  are  scored,  training  recommended,  and  exam  question  information 
stored  for  future  analysis  by  the  computer.  Test  results  are  returned 
to  the  testing  site  by  teletype.  From  the  outset  these  SAT's  have 
proved  to  be  a reliable  indicator  of  individual  knowledge.  The 
primary  measure  of  test  validity  has  been  a comparison  of  test 
results  with  the  subjective  analysis  of  the  individual  concerned  by 
their  immediate  supervisors.  No  serious  discrepancies  have  been 
noted.  The  System  Achievement  Test,  however,  is  much  more  than  a 
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vehicle  to  confirm  opinions  readily  derived  by  asking  supervisors. 

This  can  be  seen  from  the  nature  and  construction  of  the  tests. 

(Figure  1 Sample  Test  Result) 

Each  test  is  structured  to  provide  information  on  the  sub  areas  which 
make  up  an  NEC  and  to  recommend  specific  remedial  action  where  indi- 
cated. Analysis  of  individual  areas  provides  information  on  the 
effectiveness  of  training  in  each  of  the  areas. 

Personnel  Data  System 

The  Farsonnel  Data  System,  is  the  part  of  the  Personnel  and  Training 
Evaluation  Program  which  changes  a useful  testing  scheme  into  a 
valuable  evaluation  tool.  This  system  picks  up  a man  as  he  leaves 
replacement  training.  It  records  each  man's  training  and  duty  sta- 
tion history  and  provides  the  information  required  for  measuring  the 
effectiveness  of  training  at  various  locations  and  on  personnel 
with  varying  training  backgrounds.  It  also  provides  the  statistical 
base  against  which  achievement  or  lack  of  it  can  be  measured.  At 
present  there  are  over  3,000  records  in  the  Personnel  Data  System. 
(Figure  2 Personnel  Data  Sheet) 

One  can  thus  see  that  PTEP  provides  the  capability  for  collecting  infor- 
mation on  the  training  and  personnel  performance  for  which  the  FBM 
Weapons  System  Training  Program  is  responsible.  It  also  provides  a 
broad  source  of  background  material  against  which  to  evaluate  this 
information. 
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PTEP  ACCEPTANCE 


When  first  introduced  to  the  fleet.  System  Achievement  Testing  was  not 
met  with  overwhelming  enthusiasm.  Guarded  skeptism  is  about  as  close 
as  anyone  came  to  acceptance.  The  rigorous  schedule  of  examinations 
already  participated  in  by  FBH  submarine  crews  in  the  areas  of  nuclear 
propulsion  and  nuclear  weapons  safety  contributed  to  this  unenthusi- 
astic  reception. 

Gaining  acceptance  for  the  System  Achievement  Test  therefore  was  not  a 
simple  job.  A strong  public  relations  effort  was  launched  which 
started  with  the  indoctrination  on  the  FBM  Weapons  System  Training 
Program  of  everyone  taking  the  exam  as  well  as  the  administrative 
personnel  responsible  for  using  the  results.  To  further  aid  both  in 
acceptance  of  testing  and  the  use  of  test  results,  a policy  was 
adopted  whereby  only  the  command  examined  and  not  its  superiors  in 
the  chain  of  command  was  supplied  with  the  test  results.  This  gave 
each  commanding  officer  the  capability  of  evaluating  examination 
results  in  the  light  of  his  own  knowledge  of  his  ship's  training 
needs  and  to  make  judicious  use  of  the  results.  The  practice 
removed  from  both  the  commanding  officer  and  the  PTEP  organization 
the  requirement  to  defend  results  from  an  untried  tes^  instrument. 

Perhaps  the  most  important  step  toward  acceptance  was  the  procedure 
whereby  each  set  of  test  results  was  returned  to  the  com-iand  by  an 
officer  assigned  PTEP  duties.  The  results  were  discussed  with  the 
commanding  officer  and  department  heads  in  the  light  of  individual 
and  overall  crew  performance.  Carefully  kept  records  allowed  these 
PTEP  officers  to  review  trends  and  to  point  out  consistent  low  and 
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high  performers  as  well  as  anamolous  results.  The  personal  and 
individual  interest  shown  by  the  PTEP  organizations  at  the  training 
sites  effectively  sold  the  testing  system  to  the  ships. 

Training  recommendations  were  at  first  reluctantly  used  by  the  sub- 
marines. An  early  saunpling  showed  that  courses  recommended  by  PTEP 
were  used  at  about  the  same  rate  as  randomly  chosen  courses.  A 
graphical  analysis  of  successive  test  results  presented  a convincing 
picture  that  use  of  training  courses  improved  performance  in  the 
area  where  the  course  applied;  and  that  the  most  gain  for  training 
time  expended  was  realized  when  the  time  was  applied  to  weak  areas. 
This  seemingly  self  evident  conclusion,  convincingly  demonstrated 
to  management  personnel,  has  much  increased  the  utilization  of  PTEP 
course  recommendations.  The  use  of  self-study  material  recommended 
by  PTEP  was  another  area  that  gained  slow  acceptance.  As  with 
courses,  however,  PTEP  has  been  able  to  demonstrate  that  men  who  use 
self-study  material  show  improvement  in  their  overall  performance; 
consequently  the  use  of  this  training  material  has  increased 
considerably. 

ANALYSIS  AND  EVALUATION 

PTEP's  business  is  much  more  than  improving  training  utilization. 
Since  January  of  1973  sixteen  evaluations  have  been  completed  on  such 
diverse  topics  as:  curriculum  management  materials,  fire  control 

software  training,  training  of  submarine  tender  personnel,  and  the 
restructuring  of  a sonar  training  course.  Evaluation  of  test 
results  of  individuals  and  crews  have  shown  that  personal  and  leader- 
ship problems  are  reflected  in  perfoimance  trends  and  anomalies; 
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these  results  could  give  early  indication  of  personnel  problems.  It 
must  be  pointed  out  that  to  commence  and  continue  a program  such  as 
PTEP  rigorous  and  continuing  internal  evaluation  of  material,  pro- 
cedures and  current  objectives  are  necessary.  Much  attention  has 
been  focused  on  this  facet  of  PTEP.  Due  to  the  nature  of  PTEP, 
external  suggestions  for  improvement  are  not  tardy  in  coming  either. 

OPERA.TIONA.L  CRITERIA 

The  Personnel  and  Training  Evaluation  Program  described  thus  far  is 
one  which  closes  a neat  academic  loop — do  poorly  on  our  test,  take 
our  instruction  and  you  will  do  better  on  our  next  test. 

The  education  provided,  however,  is  not  an  end  in  itself,  but  a 
means  to  an  end.  That  end  is  the  effective  operation  of  the  Fleet 
Ballistic  Missile  Weapons  System.  Correlation  of  training  with 
actual  operations  is  of  paramount  importance  in  measuring  the  effect- 
iveness of  training.  Since  PTEP  can  accurately  reflect  training, 
comparing  PTEP  results  with  operational  information  should  link  the 
training  and  operating  worlds.  FBM  Submarine  patrols  and  equipment 
operation  are  meticulously  documented.  Equipment  failures  are 
analyzed,  as  are  operational  procedures  and  repair  procedures  by 
sources  outside  PTEP.  From  this  wealth  of  data,  information  should 
be  available  to  approach  criterion  testing  backed  by  operational 
statistics.  The  problem,  in  this  continuing  effort,  is  sorting  out 
the  multitude  of  dif f icult-to-relate  variables  in  order  to  correlate 
operational  information  with  testing  information.  To  date,  PTEP 
has  found  correlation  in  test  and  operational  trends.  Spearman 
Rank  Difference  correlations  have  been  positive  and  have  run  as 
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high  as  +.83.  By  July  of  1973  correlation  of  test  results  with 
operational  information  was  sufficiently  strong  to  warrant  the 
release  of  test  results  to  the  Submarine  Group  Commanders  to  assist 
them  in  evaluating  the  personnel  and  training  needs  of  their  subordi- 
nate commands.  PTEP  can  indicate  crews  with  an  increased  probability 
of  difficulty  in  handling  material  problems  arising  at  sea.  It  is  in 
no  way  able  to  predict  casualties. 

CURRENT  STATUS 

What  has  been  the  net  impact  of  PTEP  on  the  ships  supported  by  the 
FBM  Weapons  System  Training  Program.?  It  has  provided  a positive 
means  of  identifying  the  best  means  of  employing  training  time.  It 
has  ind.-cated  where  serious  training  deficiencies  exist,  particularly 
in  supervisory  personnel.  It  has  provided  a tool  for  identifying  to 
higher  commands  personnel  and  training  deficiencies  beyond  the 
immediate  control  of  the  submarine  Commanding  Officer. 

For  the  training  command,  PTEP  has  standardized  definitions  of  objec- 
tive and  insisted  upon  compliance  with  these  objectives.  For 
management  commands,  PTEP  has  dem.onstrated  the  effectiveness  and 
ineffectiveness  of  various  management  tools  built  into  the  FBM 
Weapons  System  Training  Program.  It  has  provided  a quickly  accessible 
pool  of  information  for  analyzing  crew,  rate,  or  NEC  problems.  PTEP 
has  the  capability  and  has  made  beginnings  towards  measuring  the 
product  of  tb-O  prosont  FBM  Ws2pons  Systsrn  roplsconorit 
program  against  the  product  of  the  previous  program.  Both  the 
Naval  Examining  Center  and  the  Bureau  of  Naval  Personnel  have  been 
provided  with  PTEP  products. 


PTEP  has  made  a constructive  contribution  to  the  development  and  imple- 
mentation of  the  FBM  Weapons  System  Training  Program.  Its  importance 
in  maintaining  a responsive  training  system  continues  to  grow  as  the 
Training  Program  grows.  With  the  addition  of  the  automated  access 
to  information  files,  which  is  expected  before  the  end  of  the  year, 
the  possibilities  for  analyzing  many  facets  of  the  complex  personnel 
and  system  inter-relationships  of  military  training  become  awesome. 
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THE  PERSONNEL  AND  TRAINING  EVALUATION  PROGRAM: 

A Working  Program  for  Improving  the  Efficiency  and 
Effectiveness  of  Fleet  Ballistic  Missile  Weapons 
System  Training 

Part  II  Program  Development 
by 

Frank  B,  Braun 

Data-Design  Laboratories 
Norfolk,  Virginia 

The  purpose  of  this  paper  is  to  discuss  some  of  the  problems  and 
decision  points  encountered  in  the  development  and  implementation  of 
the  Personnel  and  Training  Evaluation  Program  (PTEP) . As  in  the  case 
of  any  system  development,  not  all  the  decisions  made  in  the  heat  of 
battle  were  right  or  the  most  appropriate.  Many  of  our  solutions  may 
be  of  interest,  however,  since  we  do  have  a working  program  for  test- 
ing and  evaluating  military  technical  specialists. 

PTEP  is  basically  the  product  of  system  engineering.  The  personnel 
involved  in  the  development  of  the  program  are  engineers,  instructors, 
and  technical  specialists;  a systems  analyst  and  a programmer  applied 
automatic  data  processing  (ADP)  procedures  to  almost  all  aspects  of 
the  program. 

Since  "who"  we  were  going  to  evaluate  was  pre-determined , i.e.,  the 
FBM  weapons  and  navigation  technicians,  the  first  problem  we  had  to 
tackle  was  what  we  were  going  to  use  to  perform  the  evaluation.  We 
started  with  an  advantage,  since  a job  task  analysis  already  existed 
in  the  form  of  the  Personnel  Performance  Profiles  (PPP)  and  the 
Training  Path  System  (TPS) . The  PPP  and  TPS  provide  the  knowledge 
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and  skill  requirements  for  each  technician  in  the  program,  and  are  used 
as  the  standards  for  all  elements  of  the  program. 

We  began  our  task  by  searching  for  the  most  appropriate  types  of  test 
instruments  to  test  the  PPP  requirements.  Review  of  the  many  reference 
sources  on  the  subject  lead  us  to  the  decision  that  the  four-alternative, 
multiple-choice  question  would  be  the  most  satisfactory  for  testing 
knowledge  items.  Since  the  questions  would  be  acquired  from  many 
different  sources,  i.e.,  equipment  manufacturers  and  various  Navy 
school  instructors,  a specification  was  developed  to  insure  test  item 
standardization.  Along  with  the  question,  the  test  item  writer  is 
required  to  provide  various  information  to  relate  the  question  to  the 
PPP/TPS  standards  (Figure  1) . For  the  skill  items  contained  in  the 
PPP/TPS,  it  was  determined  that  performance  tests  would  be  the  most 
desirable  test  instruments,  but  were  impractical  for  a large  scale 
testing  program.  However,  Decision  Development  System  (DDS)  exercises, 
reported  on  at  the  1970  and  1972  MTA  meetings,  could  be  keyed  to  the 
PPP/TPS  standards  and  appeared  to  be  a satisfactory  means  of  testing 
the  technician's  application  of  knowledge.  We  decided  to  utilize  the 
DDS  as  the  skill  test  instrument. 

The  next  task  was  to  develop  tests  from  the  PPP/TPS  requirements  which 
«rould  measure  each  technician's  knowledge  and  skill  level.  The  problem 
that  became  apparent  in  this  area  was  the  wide  range  of  requirements 
for  each  technician.  In  the  case  of  the  Fire  Control  Technician,  for 
example,  there  are  28  primary  equipment  profile  tables.  10  secondary 
equipment  profile  tables,  and  two  system  profile  tables.  Each  profile 
contains  approximately  35  knowledge  and  skill  items  with  varying 
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numbers  of  sub-items  under  each  item  (Figure  2) . Obviously,  we  could 
not  test  each  and  every  requirement  within  a reasonable  length  of  time. 
Our  solution  to  this  was  the  System  Achievement  Test  (SAT)  which  con- 
sists of  both  a knowledge  and  a skill  part.  The  knowledge  part  of  this 
test  is  generated  using  a weighted  random  sampling  method.  Each  tech- 
nician's PPP  tables  are  assigned  weighting  factors  based  on  their 
relative  importance  to  the  overall  requirements.  Each  PPP  item  within 
a table  is  also  assigned  a weighting  factor  based  on  complexity  and 
importance  in  relation  to  the  other  items  in  the  table.  When  a test  is 
generated,  the  computer  uses  the  weighting  factors  to  develop  a sampling 
bank  of  all  applicable  questions.  Then, a random  generator  selects  from 
the  sampling  bank  and  generates  the  knowledge  part  of  the  test  version. 
The  skill  part  of  each  SAT  is  generated  manually  after  a thorough 
review  of  the  DOS  exercises  applicable  to  the  PPP  standards.  Each 
test  version  undergoes  a thorough  engineering  review  by  i,ystem  experts 
and  is  administered  to  a pilot  group  for  further  refining  prior  to 
issuing  as  a new  test. 

Another  type  of  test  was  developed  to  u ie  in  the  training  courses 
administered  at  the  FBM  Training  Activities  located  at  the  Guided 
Missiles  School,  Dam  Neck,  Virginia,  the  Fleet  Ballistic  Missile 
Submarine  Training  Center,  Charleston,  South  Carolina,  and  the  Naval 
Submarine  School,  New  London,  Connecticut.  This  test,  named  the 
Course  Achievement  Test  (CAT),  presented  a different  set  of  problems. 
The  PPP  Items  covered  by  each  curriculum  are  indicated  in  a chart 
which  relates  course  objectives  to  specific  PPP  items  (Figure  3) . 

This  chart,  the  Objective  Assignment  Chart  (OAC) , becomes  the  basis 
for  designing  the  CAT  for  the  training  course.  It  was  decided  to  make 
the  CAT  a criter ia-ref erenced  test  since  the  standards  could  be  defined 
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TRAINING  PATH  CHART  FOR  FIRE  CONTROL  TECHNICIAN  ( SSBN  ) POSEIDON, 

FCS  MK  88  (NEC  FT-  3306  ) (SHEET  2 OF  2)  TPC-Fl 


TABLE  INDEX 

010  FBM  Weapon  System  (System  I.evch 

100  Fire  Control  Systems  - MU  ss  Mofis  0 and  1 and  Mk  81  Mod  1 (Subsystem  Level) 

105  Alignment  Sul)system  - Mk  Mods  0 and  1 and  Mk  84  Mod  1 

lOG  Erection  Subsystem  - Mk  ss  Moris  0 and  1 and  Mk  84  Mod  I 

107  Missile  Motion  Subsv.su  in  - .MU  8s  Mods  0 and  1 and  .Mk  8 1 Mod  1 

108  Digital  Head- In  Subs\stem  - MU  88  Mods  0 and  1 and  Mk  84  Mod  1 

109  Master  CliieU  and  Timing  Subsystem  - MU  88  Mods  0 and  1 and  Mk  84  Mod  1 

110  Commanil  Sle«  Subs\  siem  - MU  s8  Mods  0 and  1 

111  Control  and  Display  Sulisvstem  - Mk  88  Mods  0 and  1 and  Mk  84  Mod  1 

112  Rate  Compensation  1 ketronics  - .Mk  88  .Mod  1 

113  Automatic  l ault  llegistration  Subsystem  - .Mk  88  Mods  0 and  1 

114  Mulliple.xers  - MU  ss  Mods  0 and  1 and  Mk  84  Mod  1 

115  Digital  Control  Comi>uter  - .MU  88  Mods  0 and  1 and  .Mk  84  Mod  1 

116  Digital  Control  Computer  to  Equipment  Configuration  Switching  and  Patching  - 
Mk  88  Mods  0 and  1 

118  Servo  Grout'  Subsystem  - Mk  88  Mods  0 and  1 

120  Fuze  Set  Subsystem  - .Mk  88  .Mods  0 and  1 and  .Mk  84  Mod  1 

121  Parallcl-To-Serial  Converter  - Mk  88  Mods  0 and  1 and  Mk  84  Mod  1 

122  Events  Translator  - MU  88  Mrxis  0 and  1 

124  Keyboard  and  Keyboard  Display  - .MU  88  Mods  0 and  1 
12G  Intercomputer  Link  - Mk  88  Mods  0 and  1 

127  Magnetic  Disk  File  - .Mk  8S  Mods  0 and  1 

131  Computer  I’rinter  - .Mk  88  .Mods  0 and  1 and  Mk  84  Mod  1 

133  Computer  Tape  Reader  Mk  \ Mods  0 and  2 - Mk  38  Mods  0 and  1,  Mk  80  Mod  2, 

and  Mk  84  .Mod  1 
135  Tape  Reader  Test  Set 

137  Power  Distribution  Subsystem  Mk  h8  Mods  0 and  1 and  Mk  84  Mod  1 
140  Test  Subsystem  - MU  88  Mods  0 and  1 

145  Test  Instrumentation  .Adapter  - Mk  s8  Mods  0 and  1 

146  Digital  Transmitter  iteceiver  - Mk  88  Mod  1 

151  Temperature  Monitoring  Power  Supply  Mk  141  Mod  1 - Mk  88  Mods  0 and  1 

155  Optical  Alignment  Group  Aik  2 .Mod  0 - .Mk  88  Mods  0 and  1 

189  Module  Test  Set  Mk  412  Mods  0 and  1 - Mk  88  Mods  0 and  1 and  Mk  84  Mod  1 
200  Missile  Systems  (Subsystem  Level) 

204  Guidance  Subsystems 

205  Gimbal  .Assemblies  Mk  3 Mod  0 and  Mk  2 Mod  1 and  Associated  Electronics 

206  Guidance  Comimter  MU  3 Mod  0 

310  Missile  Test  and  Rearliness  Equipment  .MU  7 .Mods  1,  2,  and  3 

315  Missile  Test  and  Readiness  Equii'ment  MU  (i  Mods  1 and  3 

400  Missile  I„aiinehing  Systems  (Subsystem  Level) 

500  Navigation  Subsystems  (Subsystem  Level) 

508  Multispceil  Repeater  Mk  3 \Uxl  3 

540  Frequency  Time  St.anda.rd  A.'.'  USQ  2,  2A,  and  -2D 

541  Optical  Et|uipiiunt 

600  Weapon  System  .Ship  Support  Subsystem  (Subsystem  Level) 

610  Ventilation  S\  stem 
620  Electrical  System 


Figure  2 - S/AMPLE  PPP  TABLE  INDEX 
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adequately  to  support  this  type  of  test.  Our  biggest  problem  to  date 
has  been  in  obtaining  stabilized,  approved  curricula  for  which  tests 
can  be  designed.  These  curricula  are  now  becoming  available,  and  the 
CAT  program  is  finally  gathering  momentum. 

Many  problems  arose  in  the  actual  administration  of  the  tests.  Three 
distinct  environments  existed  which  required  different  approaches  to 
administering  the  tests.  First,  we  had  the  submarine  crew  during  its 
off-crew  period.  It  was  decided  to  use  a dedicated  Navy  team  at  each 
site  to  administer  the  tests.  This  solved  the  potential  problem  of 
having  civilian  engineers  give  tests  to  military  personnel.  Every 
attempt  was  made  to  avoid  the  "Big  Brother  is  watching  you"  image  in 
order  to  obtain  the  best  possible, positive  attitude  towards  the  tests 
by  the  examinees. 

The  second  testing  situation  was  with  the  submarine  tender  personnel. 
Since  the  tenders  were  located  at  remote  sites,  test  packages  were 
developed  which  could  be  administered  by  the  tender  personnel  them- 
selves. This  meant  providing  clear,  easily  understandable  instructions 
for  each  test  section  and  directions  for  return  of  the  packages  upon 
completion  of  the  test. 

The  third  testing  situation  involved  the  CAT.  This  test  is  administered 
by  the  instructor  at  the  scheduled  time  in  the  course.  For  the  CAT,  we 
developed  a simple  proctor  guide  and  a punched,  overlay  type  ar.sv.'er  key. 
These  materials,  along  with  the  test,  are  provided  by  the  local  Navy 
teeun  to  the  instructor  on  the  scheduled  examination  day.  This  procedure 
prevents  the  instructor  from  teaching  the  test. 
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The  System  Achievement  Test,  because  of  its  objective  to  test  the 
technicians'  total  system  capabilities,  is  a relatively  lengthy  test. 

The  SAT  ranges  from  one  3-hour  session  to  four  such  sessions,  depending 
on  the  particular  PPP/TPS  requirements.  Obviously  with  a test  this 
long,  we  were  concerned  with  the  attitude  of  the  personnel  required  to 
take  it.  Surprisingly,  after  the  initial  grumblings,  the  technicians 
appear  to  view  the  SAT  as  an  integral  part  of  their  training  program. 

Some  technicians  have  even  provided  recommendations  to  improve  various 
aspects  of  the  tests. 

The  DOS  exercises  require  the  use  of  the  same  technical  documentation 
employed  by  the  technicians  when  troubleshooting  similar  casualties 
onboard  ship.  We  decided  early  in  the  development  of  the  program  that 
close  monitoring  of  the  change  status  of  the  publications  would  be 
required  to  insure  currency  of  the  tests.  It  was  also  decided  that 
the  local  Navy  team  at  each  site  should  maintain  its  own  technical 
library  to  insure  positive  control  of  the  change  status  of  the  publi- 
cations used  with  the  tests.  This  decision  created  extra  work  for  the 
teams  but  reduced  the  cries  of  "this  won't  work"  during  the  test  sessions. 

During  the  first  year  of  testing,  the  raw  results  were  mailed  from  the 
test  site  to  the  central  site  for  scoring.  This  created  a delay  of  10 
to  15  days  from  the  completion  of  the  test  to  the  time  the  results 
were  returned.  The  program  managers  in  the  Strategic  Systems  Project 
Office  directed  that  this  turnaround  time  be  reduced  no  enable  commands 
to  schedule  remedial  training,  if  the  test  results  indicated  a signifi- 
cant deficiency,  prior  to  departure  for  patrol.  An  optical  scanning 
device  and  a teletype  machine  were  used  to  solve  this  problem.  The 
Navy  team  processes  the  examinees'  answer  sheets  through  the  optical 
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scanning  machine  and  associated  data  set  into  the  computer.  The 
computer  is  used  to  score  the  tests  and  the  results  are  returned  to 
the  appropriate  team  via  the  teletype  network  (Figure  4).  Turn- 
around time  has  been  reduced  to  one  or  two  days,  depending  on  computer 
availability. 

One  of  the  largest  problems  we  encountered  in  program  development  was 
how  to  best  score  and  analyze  the  test  results.  One  goal  was  to  deter- 
mine individual  strengths  and  weaknesses  while  another  goal  was  to 
identify  and  correct  deficiencies  within  the  training  program  itself. 

We  developed  a "quick-look”  SAT  report  to  meet  the  first  requirement. 

This  report  is  based  on  the  examinees'  results  compared  to  the  existing 
fleet,  or  total,  results  contained  in  the  answer  file.  Individual 
training  recommendations  are  also  contained  in  the  SAT  Report  and  are 
based  on  Z-scorcs  greater  than  0.5  below  the  fleet  mean.  This  criteria 
was  picked  arbitrarily  but  has  worked  well  in  practice. 

A more  detailed  analysis  is  conducted  to  meet  the  second  evaluation  goal. 
This  analysis  includes  individual  and  group  evaluation  (i.e.,  submarine 
crew,  temper  crew,  instructors,  etc.),  PPP  table  evaluation,  test 
instrument  evaluation.  These  evaluations  are  conducted  after  each  test 
version  is  retired  and  replaced  with  a new  version.  The  FBM  program 
provides  a unique  environment  for  many  of  these  analyses.  Community 
idiosyncrasies  include  (1)  relatively  stable  crew  composition,  (2) 
detailed  hardware  performance  data,  (3)  close  liaison  among  all  activi- 
ties, and  (4)  extensive  personnel  history  data.  These  attributes 
enable  crew  performance  and  empirical  test  validity  evaluations  beyond 
those  which  are  normally  practicable. 
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Figure  4.  PTEP  EDP  Conununlcatlon  Network 


One  fact  became  readily  apparent  when  crew  scores  were  compared  ini- 
tially. The  differences  in  higher  and  lower  rated  technicians  in  the 
various  crews  greatly  affected  the  group  overall  score.  A method  was 
devised  to  determine  an  "expected"  score  for  each  group  based  on  what 
the  group  score  would  have  been,  had  each  examinee  scored  average  for 
his  rate.  Actual  and  expected  scores  are  plotted  on  crew  trend  analy- 
sis graphs  and  assist  greatly  in  spotting  significant  deviations. 

During  test  item  analysis,  those  test  items  whose  characteristics  do 
not  meet  established  discrimination  criteria  are  automatically  identi- 
fied for  review  and  printed  out  from  the  computer's  test  item  file. 
These  test  items  are  then  reviewed  by  Central  Test  Site  personnel  to 
determine  ir  they  should  be  deleted  or  revised,  or  if  they  provide 
an  indication  ot  a training  problem. 

We  recognized  early  in  the  PTEP  developmental  phase  that  extensive 
computer  data  piocessing  would  be  required  to  handle  the  quantity  of 
data  to  be  contained  wittiin  the  system.  Accordingly,  the  following 
five  major  con.puter  program  sub-systems  were  developed  (Figure  5) : 

1 . Test  Ceneration  Subsystem.  The  test  generation  subsystem 

incluiies  tlie  programs  necessary  to  produce  the  tests. 

a.  Test  reference  data  file  load  and  maintenance  program 

creates  and  maintains  personnel  records. 

b.  A personnel  report  program  retrieves  and  prints  out 
the  contents  of  selected  personnel  records. 

c.  A personnel  survey  program  prints  out  key  information 
from  all  personnel  records. 
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Figures  - PTEP  ADP  SUBSYSTEMS 


3.  Testing  Scoring  and  Reporting  Subsystem.  The  test  scoring  and 
reporting  subsystem  includes  programs  for  data  transmission, 
for  scoring  and  reporting  of  test  results,  and  for  update  of 
required  permanent  files. 

a.  An  input  transmission  (teleprocessing)  program  enables 
remote  transmission  of  test  data  via  optical  scanning 
equipment. 

b.  Test  scoring  and  reporting  programs  assemble  test  data, 
score  the  data  using  stored  answer  keys,  update  fleet 
norm  files,  and  prepare  reports. 

c.  An  output  transmission  (teleprocessing)  progreun  transmits 
test  result  reports  back  to  the  originating  test  sites. 

4.  Test  Analysis  Subsystem.  The  test  analysis  subsystem  manipu- 
lates accumulated  test  results  data  to  provide  information 
used  in  evaluations  of  tests,  personnel,  and  training. 

a.  Item  statistics  programs  compute  attractiveness  indices 
for  each  test  item  response,  discrimination  index  for 
each  test  item  within  an  overall  knowledge  test  part, 
and  summary  difficulty  and  discrimination  indices  by 
knowledge  area,  TOS  level,  etc.  An  engineering  review 
report  is  also  prepared  to  list  those  test  items  failing 
to  meet  a specified  value  of  the  discrimination  index. 

b.  A score  analysis  program  computes  standard  T-scores  for 
each  excuninee,  computes  actual  and  expected  average 
scores  for  each  examinee  group,  and  identifies  examinees 
and  groups  for  later,  manual  evaluation. 

c.  A skill  test  analysis  program  recomputes  skill  exercise 
scores  based  on  final  performance  data  accumulations  and 
computes  summary  statistics  for  use  in  evaluation  activities. 
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5.  Query  Subsystem.  The  query  subsystem  provides  flexible  access 
to  the  various  files  to  select,  retrieve,  and  report 

information  for  evaluation  and  management.  ! 

The  majority  of  problems  experienced  during  development  of  the  ADP 
support  systems  were  commonplace;  the  normal  growing  pains,  problems 
with  formats  and  record  contents,  and  so  on.  Perhaps  the  most 
interesting  (and  frustrating)  set  of  problems  occurred  in  connection 
with  transmission  of  test  data  from  the  test  sites  to  the  computer 
facility.  Our  equipment  provides  remote  access  via  "BTAM",  Basic 
Teleprocessing  Access  Method.  We  presumed,  as  did  the  vendors  with 
whom  we  worked,  that  the  remote  access  would  be  the  least  of  our 
problems:  you  hook  the  pieces  together,  write  a little  program,  and 

everything  works.  Unfortunately,  it  did  not  fall  together  that  easily. 

If  you  have  been  in  a similar  situation,  you  probably  have  experienced 
the  problems,  sometimes  humerous  but  more  often  maddening,  that  we  ran 
into  trying  to  obtain  information,  and  to  purchase  and  coordinate 
hookup  of  equipment  from  the  telephone  company,  the  vendors  for  the 
mark-sense  reader  and  the  data  sets,  and  the  computer  manufacturer.  It 
appeared  that  we  were  reinventing  the  wheel.  The  system  finally  jelled, 
and  today  we  have  a fairly  smooth  remote  access  capability. 

Most  of  the  PTEP  ADP  support  programs  are  fairly  typical  COBOL  appli- 
cations. One,  however,  is  not;  and  I will  describe  it  briefly.  As 
data  began  to  accumulate,  people  naturally  began  wanting  to  use  it. 

We  had  planned  to  create  a subsystem  similar  to  the  Management 
Information  Systems  so  much  talked  about  in  the  past  four  or  five  years. 
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Unfortunately,  we  could  not  precisely  define  our  information  require- 
ments: we  were  accumulating  quantities  of  data  to  which  a seemingly 

endless  number  of  questions  could  be  addressed. 

We  settled  eventually  for  a general  purpose  "query"  system.  This 
system  can  access  any  computer  readable  file  of  fixed-length  records, 
including,  of  course  the  PTEP  files.  It  works  in  three  stages,  start- 
ing with  a request  for  information: 

1.  A man  processes  the  original  question  to  define  a logical 
procedure  for  extraction  of  information  from  the  file, 

or  files,  that  contain  relevant  data.  His  function  is 
essential;  he  designs  the  algorithm  to  extract  the 
desired  information. 

2.  A COBOL  program  reads  the  algorithm  and  writes  the  imple- 
menting program  (also  in  COBOL) . 

3.  The  new  program,  which  is  unique  for  the  specific  question, 
is  compiled  and  executed,  producing  a report  tailored  to 
answer  the  original  question. 

The  system  capabilities  are  extended  by  allowing  the  output  from  one 
question  to  become  the  input  to  a second  question.  Basically,  if  we 
have  accumulated  the  relevant  data,  and  if  the  question  can  be  resolved 
to  a set  of  logic  equations,  the  query  system  can  extract  the  needed 
information.  One  major  advantage  of  this  system  is  its  flexibility. 

We  can  change  our  file  structure  around  without  losing  the  ability  to 
readily  access  all  data. 
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Looking  to  the  future,  we  are  working  on  new  analysis  techniques  which 
will  be  required  as  the  CAT  program  develops.  Since  these  tests  are 
criteria  referenced,  the  traditional  difficulty  and  discrimination 
formulas  contained  in  our  test  analysis  programs  will  not  apply. 

Several  new  types  of  test  instruments  are  under  investigation  in  the 
hopes  of  decreasing  the  testing  time  and  lowering  the  cost  of  acquiring 
new  test  vehicles.  It  is  anticipated  that  the  lessons  learned  in 
development  of  the  PTEP  for  the  Poseidon  program  will  provide  a sound 
footing  for  the  forthcoming  Trident  program. 
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learning,  vaiues  , attitudes,  abilif'es  and  progress.  Feedback  is 
essential  not  onlv  to  any  stuaent  educational  endeavor,  but  it  's 
ai.so  essential  to  the  effective  functioning  of  any  educational  system, 
i.earning  cannot  occur  without  some  form  of  feedback. 

Inc  long  range  development  program  at  ti\e  War  College  includts 
two  sep.irate  inventory  and  assessment  subprograms:  (1)  Student  inven- 
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mr.  ;or  segment  of  the  inventory  and  assessment  efforts  at  the  War  College. 

A discussion  of  student  inventory  and  feedback  immediately  brings 
to  mind  a student  "grading  system."  Traditicnally , there  has  been 
nc  f orm.1 1 i.tod  grading  academic  system  at  the  Arm?  War  College,  the 
->cnool  .It  the  top  of  the  /.rtny  school  system.  Tne  rationale  for  the 
'.onence  o.'  a grading  system  is  twofold.  First,  the  USAWC  student 
is  a m.iliirt  )rofc.ssionaj.  witlt  16  to  23  years  of  commissioned  service, 
he  I'.ns  been  selected  carefullv  on  the  basis  of  quality  of  past  perlor- 
mances  leach  year  less  than  4%  of  the  eligible  officers  are  selected  to 
attend  tne  War  College).  These  students  do  not  require  the  motivational 
impetus  ' a risotou.s  ttadinr  system.  T.-.a  second  aspect  of  the 
rattona.c  lo.  n-tt  h<iVir.,t,  grading  system  -s  that  with 
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me  group  ot  higli  achievers  there  is  obvious  and  positive  correlation 
between  grading  with  resultant  "relative  standing"  and  student  anxietv. 
The  War  College  experience  provides  an  educational  opportunity  Tor 
the  student  to  ret lect  and  learn  in  an  unstressful  environment  with 
a minimum  amount  of  student  ^inxiety — hence,  no  formalized  grading 
system  and  no  relative  stanuing  or  "l.oncr  graduate." 

Since  the  mission  of  t'ne  IS  Arm.y  War  College  is  to  educate  and 
since  learning  cannot  be  accompl islied  without  feedback,  the  task  is 
to  provide  inventorv  and  feedback  hut  at  the  same  time  minimize  student 
anxietv.  The  nevr  inventor;  and  assessment  program  attempts  to  providt- 
personal  and  confidential  feedback  to  the  stucent  without  the  inior- 
mation  being  part  of  his  official  records. 

During  AY  i'173-1974  only  nC  of  the  228  War  College  students  arc 
members  of  the  experimental  group  to  oarticlpate  in  the  new  Inventory 
and  Assessment  Program.  Four  faculty  members  have  been  assigned  to 
work  with  the  60  students  who  will  participate  in  comprehensive  personal 
inventories  and  professional  assessments.  The  assessment  is  accompl islied 
through  the  use  of  five  separate  assessment  techniques: 

a.  baseline  Inventory 

b.  Optional  Per.sonaj.  inventory 

c.  Professional  .Iclf-^tsxessmont 

d.  experiential  Assessment 

e.  Value  Gained  Inventory 

The  above  measuring  devices  cir,stit.^ce  the  basic  structure  of  the 
assessment  program.  bach  is  .^escribec  brierly  ot._ow. 

208 


I 


Base II gj  Inventory 

This  is  a baLtery  of  measures  designed  to  describe  for  students, 
developmental  coaches,  faculty,  and  the  Army  the  skills,  abilitie?., 
and  characteristics  of  the  senior  army  officer  at  a point  in  time  just 
before  he  enters  the  cast  and  aignest  echelon  of  formal  executive 
development,  the  L'SAWC.  Tae  student  must  be  given  a fairly  comprehen- 
sive picture  of  his  own  skills,  abilities,  and  characteristics.  If 
he  is  to  be  encouraged  to  develop  himself  professionally  and  personally, 
he  needs  at  least  a quasi-rat ional  basis  for  determining  which  areas 
to  work  on;  and  in  the  interests  of  attenuating  anxiety,  the  student 
nceus  to  know  liow  he  stanas  in  skills,  abilities,  and  characteristics 
in  relation  to  liis  student  contemporaries. 

A second  ancillary  rationale  for  the  Baseline  Inventory  relates 
airectly  to  the  stated  goal  of  "ta.^loring"  the  educational  experience 
to  fit  tne  inaividual  student.  The  3asei.ine  Inventory  provides,  in 
essence",  a reconnaissance  of  the  incoming  class  and  of  its  individual 
members.  The  reconnaissance  permits — assuming  a reasonable  degree  of 
pedagogical  flexibility — adjustments  in  the  curriculum  to  meet  the 
measured  needs  of  the  class,  or  of  specific  individuals.  In  tiie  ■ ase 
o.  tailoring  for  the  individual,  the  Baseline  Inventory  provides  a 
rational  basis  for  presentat  io.i  of  workshops,  and  for  "stretching" 
tne  individual  whose  wnow^^age  l.n  •.  given  area  eCi.als  or  exceeds  that 
of  the  fac’alty. 

Tiie  methodology  used  in  the  B<_seline  Inventory  assesses  the  indivi- 
dual's (anc,  when  aggregated,  the  class's)  ^cve^  of  competence  in  a 
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variety  ot  areas  related  directly  or  indirectly  to  the  curr iculuir. . 


The  inventory  can  answer,  for  example,  a question  such  as:  "Does 

this  student  Know  tiie  field  of  human  relations  well  enough  to  write 
a book,  or,  are  the  principles  and  terminology  of  the  field  so  unfami- 
liar that  he  must  exert  extra  effort  to  read  or  discuss  the  course 
material ?" 

The  Baseline  Inventory  can  achieve  several  inventory  and  assess- 
ment objectives.  The  student,  comparing  his  individual  profile  to  the 
competency  range  envelope  of  the  class,  has  a rational  basis  for 
planning  his  own  program  for  professional  and  personal  development 
(e.g.,  attend  workshops  for  "weak"  areas).  The  student's  counselor 
or  coach,  studying  the  profiles  of  his  counselees,  has  a more  objective 
basis  for  understanding  and  coaching  his  counselees  as  unique  individuals. 
ITie  faculty,  studying  the  group  profile  and  comparing  it  with  baseline 
measures  of  previous  student  classes,  has  an  objective  tool  for  curri- 
culum modifications  which  recognize  difference  in  classes  from  year 
to  year.  'SAWC,  studying  individual  profiles  can  accomplish  (with 
computer  support)  several  desirable  objectives:  (1)  tailor  student 

committees  toward  equivalent  skill  composition;  (2)  identify  students 
who  are  weak  in  a number  of  areas  and  require  careful  coaching  programs; 

(3)  identify,  for  recruitment  as  permanent  faculty  or  for  use  as  associate 
instructors,  those  students  wno  have  exceptional  strengths  in  areas 
directly  related  to  the  USAWC  curriculum  or  mission;  and  finally, 

(4)  provide  for  the  Officer  Personnel  Directorate  a special  "assignment 
potential"  profile  which  indicates  those  areas  where  the  individual 
nas  .strong  or  except icr.a...  coT;ett..cl<.£  (cr.lj'  ...'.on  e^.pressiy  aesired  by 
tne  studeTit)  . 
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Th<_-  firsL  Baseline  Inventory  was  :r, ailed  to  A'.'  7 A students  early 
in  the  year  prior  to  the  time  they  reported  to  the  College.  The 
inventory  was  scored  and  returned  to  students  as  "Feedback."  Addi- 
tional Iv  the  r.icultv  was  provided  with  pertinent  data  as  a group  result 
ol  tile  inventory  to  obtain  a class  proille. 

Ta.ere  are  several  recopnizea  shortcomings  of  the  Baseline  Inventory 
instrument.  Both  are  related  to  the  fact  that  the  instrument  is  a 
self-inventory — the  individual  assesses  his  own  levels  of  skill  in 
areas  as  defincu  by  the  instrument.  We  can  assume  that  some  individuals 
will  intentionally  overstate  their  own  capabilities.  We  can  assume 
also  that  some  individuals  will  misinterpret  or  not  fully  comprehend 
the  implications  of  tf.e  area  aescr ipticns.  These  possibilities  are 
iiorne  in  mind  when  interpreting  *.idividual  profiles.  However,  for 
aggregate  profiles,  tiic  shortcomings  are  less  s i f.nif  leant  'oeca;ise 
.■rrors  occur  in  both  directions;  e.g.,  some  IndiViGuals  will  understate 
intentionally  their  own  ab,.l^tles.  The  variations  also  provide  the  coach 
with  one  indicator  of  the  student's  self-concept. 

Further,  wlille  we  have  thus  far  (for  purposes  of  illustration) 
aiscussed  the  Baseline  Inventory  as  a single  instrument,  we  should, 

;n  reality,  consider  it  as  ser..es  of  related  data  collection  tests 
and  surveys.  Initialry,  we  began  with  the  Base_ine  Inventory  designed 
around  curriculum  objectives,  dver  time,  we  will  substitute  validated 
and  estaDiisneci  measureme.'.t  inscrumencs  for  portions  of  the  Baseline 
inventory.  The  Graduate  Aeccrc  Frxamir.aticn  (GR.E)  can  be  used  to  assess 
generai.  academic  competency.  Tne  Miller  .Analtgies  Test  can  be  used 
iCr  a i3n_rc.  -r_  t.  'i. Ci...;  . Trere  are  innumerable 
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LosLs  available  for  measuring  raading  comprehension,  listening  compre- 
hension, and  writing  skills.  And  the  Educational  Testing  Service,  given 
clear Iv-stated  learning  objectives  and  supporting  course  material, 
can  develop  a GRIi-like  comprehensive  examination  with  alternate  forms, 
based  on  the  foundational  elements  of  the  USAWC  curriculum — ideal  for 
a "before-and-af ter"  measurement. 

Optional  Personal  Inventory  (OPI) 

Tlie  second  major  assessment  task  has  as  its  primary  objectives 
the  personal  development  of  the.  individual.  Good  research  on  executive 
development  suggests  strongly  that  the  point  in  time  near  age  40  and 
two-thirds  through  a career  is  a milestone  point  for  contemplation  and 

ad justment/af f irmation  of  personal  and  professional  goals  and  modus  | 

operand  i . The  Optional  Personal  Inventory  (OPI) , supported  by  the  ' 

coaching  program,  is  designed  to  assist  tne  student  in  this  difficult 
introspective  task.  Tlie  m.ethodology  of  the  OFI  is  far  less  structured 
tnan  that  of  the  Baseline  Inventory.  IVhereas  the  Baseline  Inventory  is 

1 

conducted  prior  to  the  student's  arrival  at  the  USAWC,  the  Optional 

I 

Personal  inventory  is  dependent  for  its  success  upon  a sound  student- 
coach  relationship  and  cannot  commence  until  such  a relationship  has 
been  initiated.  A battery  cf  inventories  a.»-e  available  to  students 
on  .an  optional  basis.  incjluaec  in  tne  cattery  ace  instruments  to  mea- 
siice  .'ittituaes,  speaking,  csi.cl.ig  and  welting  ahilitv,  personality,  need.s , 
vocational  interests,  etc.  initial  ccacning  sessions  with  incoming 
students  covered  the  purpose  of  the  in, ar.wories  and  stressed  the  most 
appropriate  time  lor  making  clear  to  the  stiic'e.'it  tnat  die  OPI  program 
opiio'.ii*  ar.c.  t..at  h.  „a..  . ..  ,•  time.  era.-/  trer.  t.'. e progran. 
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There  are  two  sources  of  evidence  which  suggest  the  strong 
potential  of  the  OPI  concept. 

In  the  executive  development  field,  one  of  the  most  promising  and 
rapidly-growing  methods  of  developing  executive  potential  at  higher 
levels  is  the  assessment  center.  Executives  visit  these  centers  and 
participate  in  an  "assessment  week,"  wherein  measures/observations 
are  made  of  various  skills  and  characteristics  which  have  been  found 
to  be  closely  related  to  executive  proficiency.  An  intensive  "feed- 
back" session  follows  the  assessment  week.  During  the  feeabac:;  session 
the  executive  is  given  objective  and  professional  interpretation  of 
the  measures  made  during  assessment  week.  This  is  privileged  infor- 
mation; the  staff  assessor  is  not  a memoer  of  the  parent  organization; 
and  the  individual  executive  is  free  to  do  what  he  wishes  with  the 
feedback  information  he  is  given  (written  and  grapnic  maceriaj..  audio 
and  videotape  cassettes).  The  US  Army  investigated  the  potential  o-f 
these  centers  in  1971-72.  As  a result,  more  than  a dozen  general 
officer  designees  have  attended  assessment  centers  last  year;  a. id 
plans,  personnel,  and  funds  for  establishment  of  several  Army  assess- 
ment centers  ..ave  oeen  approved. 

The  second  support  for  c.ie  OPI  concept  comes  directly  from  the 
experience  of  approximately  15  JSAWC  students  w.io  part  i-cipa.ed  in  a 
Personal  inventory  and  Atialysis  program  in  A'f  72.  Tnese  15  students 
were  enrolled  in  a graduate-level  counseling  program  at  nearby  Snippens 
burg  State  College.  They  all  participated  in  a wide  variety  of 
assessment  tests  and  situations;  then  each  teex  all  of  the  cata 
per tai.-.j.r.g  to  his  ~vt.  per ftr'...a’.;ca  ana  aid  a co.i.,...etu  cc -nseii..^ 
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"work-up"  (scoring,  interpretation,  prognosis,  prescription)  of  the 
available  data.  'Hie  student  group  was  highly  enthusiastic  about  this 
in-de|-th  objective  exercise  in  self-development,  'iliere  was  virtually 
unanimous  agreement  that  it  was  extremely  valuable  ii.  terns  of  persona^ 
;ind  ,1  ,'o  f es s iona  1 devc  1 opnen  t . 

Prote--.s  : onal  Self-Assessment 

R<ationale  for  this  as.ses.sment  exercise  springs  .iirectiy  from  the 
law  of  learning  noted  earlier,  i.e.,  w^Lhout  knowledge  of  results, 
attempts  to  learn  are  only  random  behavior.  Therefort  the  purpose  of 
professional  self-assessment  is  to  individualize  feednack  to  the  stucunt 
regarding  liis  attempts  to  learn  ^ and  his  personal,  profess lotia.  and 
leadership  developmert ; to  provlae  feeuback  to  the  instructor  regardinc 
111-,  teaching;  and  to  provide  rich  source  of  empirical  data  for 
continued  curriculum  improvement. 

bsaenria^ly , in  the  profcssicnal  self-a.ssesstnent , ^ne  metl.odology 
requires  no  change  in  Lite  present  greup  requirement  wiiich  accompanies 
each  course.  In  addition  to  the  o''Oa?  tequrrement,  i.owev.r,  tiiete  shou'ci 
be  an  individual  requirement.  At  or  near  the  end  of  the  coerse,  tne 
student  is  g^ven  an  examir.at  .or.  whicr  ontair.s  s,.bjective  c ^es  t i^a-.s 
requiring  responses  at  revels  ol  knovieuge  a'  i.  co.ttent  area?  -ert'oe. 
fro:;,  tne  stated  course  objectives. 

. r. -*  sc  iciti'.k.  is  given  t./c  <\v..ui*s  s.o  ckv-u*  est  '.w.,..  nt  c.  i 

:iis  vtr.swers  car.  dc  i.r*  any  .orr.kiC;  wr*.:e.  ourlinta, 
topical,  or  grcipl'.icui.  .At  thv  end  of  tnis  ..irr.t  .le  a.  viu«.->-  with 


tUC.k.C'ir. 


- ' a xrer  t 


. k,  ..  . . .C  ...k«  ...  . r.kw  . C tiki  J . . 
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an  official 


and  who,  if  presented  with  tne  examination  questions  ir. 
capacity,  would  respond  as  indicated  on  the  solution  sheet. 

The  student,  by  sex f-assessment , compares  his  answers  with  those 
on  the  solution  sheet.  He  is  tiien  given  the  opportunity  to  discuss, 
debate,  or  support  the  compc  .ents  of  ti'.e  expert  solution — preferably 
with  the  expert  himself. 

From  this  comparison  and  discussion  process  there  undoubtedly 
will  come  valid  ir.dlviauai  .tlnority  views  and  creative  comment.  These 
should  be  encouraged  (but  net  in  class)  and  can  serve  as  a basis  for 
subsequent  individualized  discussion  between  student  and  faculty. 

At  the  completion  of  the  examination-comparison-discussion 
process,  the  student  turns  his  paper  in  to  the  faculty  member.  Tuere 
is  a clear  stipulation  that  no  paper  be  identified  with  the  individual 
student's  name.  (An  "option"  on  signing  examination  papers  would  have 
coercive  implications).  The  purpose  of  this  turn-in  of  examination 
papers  is  to  provide  the  faculty  with  empirical  subjective  feedback 
on  how  well  learning  objectives  were  achieved. 

Experient-i.al  Assessment 

Experiential  assessment  consists  of  three  separate  assessment 
tasks  conducted  wliile  the  student  is  devoting  his  entire  educational 
efiort  tu  p .rt  icipation  in  r.  itoup  problem-solving  exercise.  The 
assessment  . r cnis  phase  termed  "experiential  assessment"  becavise 
tiie  research  objective  ana  the  group  problem-solving  method  represent 
a model  of  ^r.a  professional  requirements  tnac  the  student  will  most 
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probably  face  foliowing  L'SAlvC  gi'c-Juation.  With  some  exceptions,  the 
jU'of ossiona j requirements  which  he  is  being  prepared  to  meet  will 
involve  working,  lor  a s uperord 1 na t e leader  or  director,  as  a member 
ul  a group  (slat!  or  committee),  and  producing  a product  wnlch  must 
withstanu  the  scrutiny  of  experts  in  the  f-eld.  The  th.rej  assessment 
tasks  of  this  phase  arc  geared  to  the  assessments  the  student  w’ili 
encounter  in  his  post-graduate  environment;  i.e.,  those  of  has  superior, 
his  peers,  and  his  critics.  The  team  leaner  or  project  director,  pre- 
pares an  assessment  of  the  indiv-,duai  student  which  would  be  somewhat 
similar  to  the  Officer  Efficiency  Report. 

A second  task  of  experiential  assessment  seeks  to  provide  the 
student  witii  feedback  pertinent  to  his  performance  as  a .nemner  of  a 
.,;-am.  as  seen  by  his  coileagnes  or  peers.  Agai-n,  the  polic.y  ol  as.->uted 
anonymity  will  appj.y,  and  the  student  hi  1 ms e 1 f is  to  be  the  solo 
i_ustom.er  for  the  peer  c.ppraisai  information. 

The  third  task  of  expvrientiai  assessment  is  apprai  .al  liy  sub’c-ct 
area  experts.  This  third  assessment  task  has  its  major  ;.r.p..ct  in  the 
motivational  area  rather  than  t .e  dcvelopme..tal  area.  From  tae  outse.. 
of  t'lie  project,  researcii  tcan.  members  w.  li  know  .hat  taeir  group  p^OjCct 
to  oe  evacuate' d oy  an  cu.-S.u*'^  .'■■xpCi.  tae  liej.d,  engaged  spec*ticaii^ 
to  function  as  .r.tic  of  t.h.:  s-‘Ov.p  re.-.ciarca  product. 

reedbacK  fro.r  ....c  tnree  t.o.sessnet  to  of  tnt  .xper  ient:,ai  Assessment 
task  s.ncui.d  provide  the  stuuert  wit.i  clear  a.iG  aetailed  insight  of 
wiiat  he  should  do  to  funct.t.;  vios;  efiectively  ir.  the  proolem-sol’'ing 

c.'!*'.’ i rcr...v.wn w r;ci  ±s  .r.vjw  ' “o  racc  ci' 
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Va  1 u e inventcry 


We  have*  good  evidence  that  change  occurs  within  the  student 
during  the  I'SAWC  year.  We  assume  that  changes  occur  also  in  otiier 
areas  — in  functional,  academic,  geographic,  and  interpersonal  abilities, 
for  example,  but  we  are  nard-pressed  to  say  just  what  has  changed, 

and  iiow  mucii,  and  in  wnat  direction.  ! 

The  assessment  task  of  the  Value  Gained  inventory  is  an  attempt 
:o  aeterr.iiie  tne  total  developmental  effect  (upon  the  student)  of  the 
ye.'.r  spent  at  the  US  Ar.r.y  War  College.  The  student  needs  this  infor- 
mal; ion  ; n order  to  review  ann  appreciate  the  professional  and  personal 
ueveiopment  brought  aoc.*t  ry  cht;  total  educational  experience.  Suen 
a revie*  appears  prerec;uisi>.e  to  i.idividuai.  planning  for  continuing 
acveloprr.ent , extencing  far  oeyond  liSAWC  graduation. 

The  Value  C'.ained  Inventory  ts  even  more  .critical  to  the  War  College 
itself.  W'e  need  a year-end  summary  assessment  to  determine  the  degree 
to  which  our  educational  system  nas  achieved  its  curricular  objectives. 

We  r.eec  facts  to  answer  with  some  nrecisio.i  tiie  of teu-asked  quesuio', 

’'What  sort  of  man  does  the  J.SAWC  nroduca?" 

The  methodology  of  the  Va^ue  Gained  Inventory  consists  simply  of 
.ic.r.anister  .ng  alter.iate  anu  eq.^va^ent  forms  of  tuie  instrumients  used  i 

■at  tile  ofcgi.-r.^r.g  of  the  ...cauui.ic  year,  woula  he  sufficient  time  for  1 

f.it  uat.i  ..jf  fcticn  requ-re.r.e.', ts  oi  tne  Value  Cal.-ied  Inventory.  Most 
of  whe  ir-.cruments  will  oe  co.r.pc.'.en _s  of  tr.e  Ease.ine  Inventory,  but 
measure.s  from  tus  Optional  fernonai  Inventory  couiu  be  induced. 
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Statistically  significant  uiifcrences,  on  pairings  of  data  for  the 
same  individuals,  strongly  suggest  effects  directly  attributable  to 
llie  USAWb  vear. 

: . umma  r v 

T'ne  program  of  student  inventory,  assessment  is  a trial  program 
conducted  on  an  exper iemtna i basis  for  Academic  Year  1973-1974. 
after  ..  comprehensive  evaluation,  the  program  is  determined  to  he 
worthwtiile,  it  may  be  modified  to  include  a larger  segment  of  the 
.students  at  the  .Army  War  College.  Those  portions  of  the  progr.'.m 
deter. sined  to  be  of  marginal  value  will  be  eliminated. 

During  the  conduct  of  the  program  certain  basic  principles  must 
he  :ol lowed.  First,  the  student  participants  must  realize  the  pro- 
cr.t."'.  c;  oriented  toward  hel-.i.'ig  them  sharpen  the  edges  of  their  own 
.■ikiii.s,  le.arning,  value.s,  and  knowledge.  They  are  not  considered 
p.itiencs  or  uniieaithy  individuals,  but  successful,  mature,  proven 
executives.  An  analogy  can  be  made  of  their  career  performance  with 
that  c‘  a mar.tsman  rifle  range.  Thus  far  in  their  career,  they  have 
already  fired  "expert."  But  until  this  time  the/  have  not  always 
knovTi  wlie.re  the'’r  shot  groups  have  been  located  on  the  targets.  Thc- 
: acuity  coach  h.as  tne  function  o'  marking  some  of  their  targets.  i’orliaps 
w'iti'.  a minimum  am.ount  cf  coac.a.it;  the  s.not  group  can  he  moved  to  the 
cer.:er  of  th.c  huiiseye.  With  th.^  proper  coaching  even  an  expert  can 
improve  his  "score." 

The  second  priiciple  is  that  of  confidentiality.  The  student 
oarticipanr  must  be  assureu  f.a^  .icne  of  the  aata  gat.iared  will  he 
. tr  ip  a.i ..  _t  ir...  i^rsc.'.a..!'’  I .ta  inforriatio  i ..'.usi  not  he  u..^d  ,r.  any 
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official  records  or  reports  which  identify  tne  individual.  The 
relationship  between  the  student  participant  and  the  faculty  coach 
must  be  a privileged  relationship. 

The  other  principles  e.g.,  honesty,  empathy,  respect,  concreteness, 
self-disclosure,  and  immediacy,  of  an  effective  helping  program  are 
also  important.  Since  the  program  is  on  a trial  basis  this  year  there 
is  some  skepticism,  and  apprehension  by  a few  of  the  observers.  For  u'.e 
most  part,  everyone  participating  in  the  inventory,  assessment  and 
feedback  effort  is  enthused  and  confident  it  will  enhance  the  educa- 
tional endeavors  at  the  United  States  Army  War  College. 
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SUBORDINATE  RATINGS:  WHY  NOT? 
W.  H.  Githens  and  R.  S.  Elster 
U.S.  Naval  Postgraduate  School 


The  notion  of  having  subordinates  rate  their  seniors  is  one 
that  typically  engenders  a great  amount  of  feeling.  These 
authors  became  interested  in  the  topic  while  teaching  graduate 
level  psychology  courses  to  military  officers  when  we  found 
that  a quick  way  to  provoke  class  discussion  was  to  assert 
that  a system  for  gathering  subordinate  evaluations  of  seniors 
seemed  deserving  of  experimentation.  Over  the  last  several 
years,  therefore,  we  and  a number  of  our  students  have  con- 
ducted some  studies  concerning  subordinate  evaluations.  It 
is  to  these  studies  that  we  will  now  turn. 

TWO  GENERAL  SURVEYS  CONCERNING  SUBORDINATE  RATINGS 

In  1971,  Lt.  J.  G.  Bloomer,  USN,  working  with  one  of  the  authors, 
sent  1100  survey  questionnaires  to  the  U.  S.  officers  attending  the  Naval 
Postgraduate  School.  Over  350  officers  responded  to  this  survey  con- 
cerning subordinate  evaluation.  Lt.  Bloomers'  survey  focused  on  the 
acceptance  o.f  subordinate  evaluations  when  they  are  for  the  senior's 
use  only.  Tims,  his  instructions  read,  in  part,  "...The  purpose  of  this 
survey  is  to  discover  whether  the  students  of  NTS  feel  that  the  military 
officer  might  be  benefited  by  a program  such  as  this,  in  which  he  is 
evaluated  by  his  subordinates,  with  the  ratings  submitted  directly  to 
him  solely  for  his  own  use.  The  responses  obtained  to  this  survey  are 
summarized  in  Table  I. 


TABLE  I 

Results  of  Lt . Johm  Bloomer's  Survey 
of  Naval  Postgraduate  Students 
Attitudes  Concerning  Subordinate 
Evaluations  (N=350) 

1.  .'o  you  feel  such  a periodic  survey  by  his  enlisted  men 

would  be  a significant  help  to  the  junior  officer? 

Yes  ^laybe If  no,  go  to  part  111. 

Note:  Lt.  Bloomer  grouped  the  number  answering  "Yes  and  "Maybe" 

into  "those  finding  some  merit  in  the  program,"  since  in  the  majority 
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of  cases  the  "Maybe"  answer  seemed  to  indicate  concern  with  the  mechanics 
of  the  program,  not  necessarily  the  value. 

Total  finding  some  merit  in  the  program: 


Overall  (all  officers  responding)  74 . 5Z 
LT  and  below  83 . 2% 

LCDR  and  above  68 . 0% 

Army  (of  10  responses)  80.0% 

Marines  (of  22  responses)  45.5% 


2.  Should  sncli  a program  be  restricted  to  the  junior  officer  only? 


Yes  15.4% 


No  83.6% 


Note;  The  most  common  response  to  "level  it  should  be  restricted 
to"  was  all  below  flag  rank;  however,  many  felt  such  feedback  would  be 
helpful  all  the  way  to  CNO. 

3.  Do  you  feel  that  such  a system  would  liave  a significant  effect 
on  enlisted  morale? 


Yes  61.5% 


No 


15.8% 


Don’t  know  22.7% 


4.  If  you  were  rated  by  your  subordinates,  would  you  consider 
these  reports: 


Quite  seriously  33.4% 
Seriously  57 .4% 


90.8% 


Casually  3 . 9% 
Other  5.3% 


5.  If  you  found  such  reports  adverse,  would  you  attempt  any  changes 
in  your  leadership  techniques? 


Yes  72,7%  No  0 Don't  know  27 . 3% 

Note;  This  question  elicited  many  comments,  and  in  most  cases  the 
"Don't  know"  answer  was  a qualified  "Yes".  That  is,  after  consideration 
of  the  source,  how  tlie  program  was  run,  whetlier  they  were  isolated  reports 
or  a trend,  etc,,  a leadership  style  change  might  be  attempted. 

6.  Do  you  feel  tiiat  Commanding  Officers  might  gain  by  having  a 
similar  evaluation  made  of  them  by  their  junior  officers? 

Yes  76.4%  No  14,0%  Don't  know  9.6% 
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Note:  It  is  interesting  to  note  that  many  who  had  held  command 

commented  that  this  feedback  would  have  been  especially  useful  to  them. 

7.  Do  you  feel  that  such  feedback  might  have  been  of  help  to 
you  at  any  time  in  your  career?  If  yes,  when? 

Yes  87.9%  No  12.1% 

Note:  The  "when"  in  this  case  was  generally  as  a LT  and  below, 

but  many  responded  that  it  would  help  at  any  time. 

8.  Do  you  feel  that  enlisted  raters  would  conscientiously  attempt 
an  accurate  evaluation  of  their  superior  officer? 

Yes  74 . 2%  No  7 . 2%  Don't  know  17.6% 

9.  If  such  an  evaluation  system  were  adopted  by  the  military 
services,  do  you  feel  that  the  raters  should  be  restricted  to: 

No  restrictions  (all  levels  of  subordinates)  58 . 3% 

Petty  Officers  only  34 .6% 

Other  7.1% 

Note:  The  "Other"  in  this  case  generally  was  accompanied  by  comments 

suggesting  a graded  grading  system;  that  is,  officers  and  CPO's  grade  CO  & 
XO  while  Chiefs  and  First  Class  rate  their  Department  Head,  or  some  similar 
groupings.  There  is  certainly  some  merit  to  this  idea,  if  only  to  reduce 
the  amount  of  paperwork. 


file  results  of  this  survey  appeared  to  indicate  two  points  conclu- 
sively: We  quote  Lt.  Bloomer,  "(1)  Tliere  is  a general  concern  for  improving 

officer/enlisted  lines  of  communication.  (2)  The  overwhelming  majority  of 
respondents  indicated  a high  degree  of  respect  for  the  judgement  of  the 
enlisted  man  of  today."  Ihe  reader  should  keep  firmly  in  mind,  liowever, 
that  Lt.  Bloomer's  survey  had  as  a m.ijor  premise  tliat  tlie  subordinates 
evaluations  would  be  for  the  use  of  only  their  senior. 

At  about  thi'  same  time,  Lt . Bloomer  was  conducting  his  survey,  Githens 
surveyed  68  USN  ofticers  al  i iie  Navy  Postgraduate  School.  Two  of  his 
questions  involved  t lie  issue  ol  ratings  by  subordinates.  The  questions 
and  the  responses  obtained  are  presented  in  Table  II.  The  reader  should 
notice  that  in  tlie  instructions  to  Githen's  survey,  unlike  those  to  Lt. 
Bloomer's  survey,  the  use  of  the  data  from  subordinates  ratings  were  not 
explicitly  restricted  to  use  only  by  tlie  senior. 
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TABLE  II 


Results  of  a Survey  by  W.  H.  Githens  of  68  USN 
Officer/Students  at  the 
U.S.  Naval  Postgraduate  School 


As  part  of  an  overall  evaluation,  what  do  you  think  about  being 
rated  by  your  subordinates? 


An  excellent  idea  _3 

A good  idea 

A fair  idea  1_£ 

A poor  idea  ^ 


Assuming  that  peer,  subordinate,  self,  and  the 
by  superior"  were  in  operation,  what  should  be 
to  the  total  evaluation?  Assign  a weight  from 
system.  The  sum  of  the  weights  must  add  up  to 


present  "rating 
their  contributions 
0 to  100  each 
100  units. 


Rat ing 

by 

peers 

13.87* 

Rat ing 

by 

subordinates 

8.00* 

Rating 

by 

superiors 

72.23* 

Rating 

by 

self 

5.79* 

* Means  computed  over  sample  of  68  officer/students. 


The  results  of  Githen's  survey  seem  to  show  that  officers  are  less 
sanguine  about  subordinate  ratings  tlian  did  the  results  of  Lt . Bloomer’s 
study.  We  feel  that  this  difference  in  results  stems  from  the  psycholog- 
ical sets  given  by  the  questions  and  instructions  to  the  surveys.  Lt. 
Bloomer's  survey  addressed  using  subordinate  ratings  as  a feedback  vehicle 
for  the  cognizant  senior,  while  Githen's  survey  asked  the  respondents 
about  using  data  from  subordinate  ratings  when  evaluating  officers.  These 
results  lead  to  the  rather  obvious  conciusion  that  the  use  to  which  sub- 
ordinate evaluations  are  put  will  influence  the  reception  they  receive 
from  those  who  are  evaluated.  Thus,  it  is  necessary  to  understand  the 
fears  and  concerns  that  officers  have  concerning  subordinate  ratings. 


In  his  survey,  Lt . Bloomer  asked  those  who  were  opposed  to  the 
type  of  subordinate  rating  system  he  proposed  to  comment  on  their  reser- 
vations. Table  III  contains  a representative  set  of  t..e  comments  that 
were  obtained. 
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TABLE  111 


Representative  Comments  of  Officers 
Opposed  to  a Subordinate  Rating  System 
as  proposed  by  Lt.  Bloomer 

LT  "This  system  of  evaluation  would  turn  into  a popularity  contest 

between  junior  officers."  Note:  This  comment  was  the  most 

common  reason  given  for  tlie  negative  response. 

LT  "1  believe  face-to-face  discussions  are  more  productive." 

CDR  "Undermines  discipline.  A crutch  for  poor  leaders  who  don't  have 
the  ability  to  communicate  with  men,  or  to  accurately  comprehend 
wliat  is  going  on  around  them." 

LT  "It  is  difficult  enough  aboard  ship  to  instill  in  a junior  officer 

that  the  principles  of  good  management  are  best  served  by  a 
professional  relationship  with  his  men  rather  than  a friendship 
basis.  This  evaluation  would  only  serve  to  promote  the  latter." 

LCDR  "There  is  just  too  much  danger  in  the  proposal  for  all  concerned 
for  the  few  benefits  that  might  be  realized." 

LCDR  "A  good  officer  knows  how  his  men  feel  about  him.  The  officer 
who  dues  not  know  is  unlikely  to  be  affected,  pro  or  con,  by 
written  evaluations." 

LCDR  "There  is  enough  breakdown  in  authority  in  the  modern  Navy  without 
open  criticism  by  subordinates." 

LCDR  "1  feel  peer  group  pressure  or  influence  would  greatly  prejudice 

t!ie  non-rated  opinion.  Senior  Petty  Officers  would  be  more  capable 
of  fair  evaluations." 

LT  "The  main  reason  I said  no,  is  that  1 feel  Liie  evaluation  would 

be  degraded  to  a grievance  session..." 

LCDR  "Personally,  such  a system  in  the  military  service  goes  against 
my  grain." 

LCDR  "God,  when  is  tills  liberalism  ever  going  to  cease?” 

LT  "Onoe  comments  are  noted  on  paper,  the  'Navy  wav'  ensures  the 

CO/XO,  Dept,  will  manage  to  see  them." 

( 

LT  "strongly  1 eel  that  feedback  should  be  confidential  and  not 

available  to  superiors." 
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LCDR 

LCDR 

LCDR 

LCDR 

LCDR 

LCDR 

LT 

CDR 

LT 


"I  feel  a junior  officer  is  very  formable  in  his  early  years  and 
should  be  guided  only  by  his  seniors,  not  his  juniors." 

"The  evaluation  would  seem  to  put  too  much  emphasis  on  molding 
the  junior  officer  to  suit  the  needs  of  subordinates,  when  in 
fact  the  molding  should  come  from  the  top  — providing  of  course 
that  the  superior  is  capable  himself  and  takes  advantage  of 
opportunities  to  observe  the  junior  officer." 

"One  would  tend  to  cater  to  the  wishes  and  whims  of  subordinates 
if  only  to  protect  his  ego  when  he  receives  their  'biased' 
evaluations . " 

"Feedback  should  be  welcome  in  any  form.  It  enables  one  to  see  if 

points  of  emphasis  are  coming  across.  it  should  be  entirely 

optional  on  tlie  part  of  the  enlisted  men  and  should  not  go  through 
any  cliain  of  command,  i.e.,  leading  PO's  or  Chiefs." 

"The  typical  junior  officer  is  too  sensitive  to  the  opinions  of 
his  subordinates.  By  trying  to  please  all  of  them,  having  no 
way  to  differentiate  between  serious  comment  and  sarcastic, 
he  could  ruin  himself  in  his  ovni  image.  He  could  lose  whatever 
self-confidence  he  had  built  up." 

"I  feel  the  success  of  such  leedback  would  depend  upon  wdiether 
the  recipient  could  take  criticism,  and  if  he  could,  effect 
changes  where  necessary.  There  is  a danger  of  an  impersonal 
system  sucli  as  that  described  becoming  a routine,  meaningless 
exercise  if  the  recipient  cannot  communicate  its  worth  by 
viewing  the  feedback  seriously  and  making  clianges  where  necessary." 

"Problem  in  maintaining  anonyTnity  comes  up  when  the  officer  lias 
very  few  subordinates.  May  need  to  propose  a system  in  wliich 
evaluations  are  only  made  when  the  otficer  iias.  Cor  example,  five 
or  more  subordinates."  Note:  Some  form  ot  this  comment  was  often 

made  and  it  is  probably  true  tliat  maintaining  true  anonymity  in 
a small  unit  would  be  difficult,  it  not  impossible. 

"The  evaluation  should  not  be  forced  upon  the  subordinates.  It 
should  be  optional  whether  or  not  you  evaluate  your  senior." 

"Since  seniors  have  a hard  time  evaluating  juniors  and  our  fitness 
reports  are  a highly  controversial  topic,  don't  you  think  an 
uneducated  individual  would  have  even  a more  difficult  time 
efficiently  evaluating  seniors?' 
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LCDR  "I  don't  feel  a PO  is  qualified  to  tell  me  how  to  do  my  jobi" 

LT  "l  do  not  believe  that  enlisted  men,  in  general,  have  the  back- 

ground to  evaluate  my  performance,  since  personnel  management 
is  only  part  of  my  job  as  a Naval  Officer.  Most  could  not 
evaluate  my  shiphandling  ability  since  most  have  no  experience. 

They  could  not  evaluate  my  material  and  financial  management 
since  they  do  not  have  the  training  I have,  etc.,  etc. 

LT  "1  do  not  feel  most  younger  PO's  and  non-rated  men  would  be  honest 
and  objective  in  making  out  such  evaluations.  Also,  I feel  it 
would  be  difficult  to  keep  these  evaluations  from  the  concerned 
officer's  superiors." 

LT  "I  believe  that  the  average  sailor  working  for  a young  JO  is  not 
mature  enough  to  make  the  type  of  comments  which  would  be  bene- 
ficial ." 

LCDR  "1  think  the  whole  idea  is  a bunch  of  ! If  you  can  subject 

the  EM's  to  the  same  pressures,  hold  them  to  the  same  responsibi- 
lities, etc.,  THEN  they  are  in  a position  to  rate  him  objectively, 
and  not  before." 

LCDR  "A  good  manager  should  be  able  to  evaluate  his  policies  by  the 

results  of  his  efforts  or  by  another  means  of  feedback.  Subordinate 
evaluations  <ire  a nice  tliought  but  probably  useful  only  if  the 
superior  has  really  lost  contact  with  his  subordinates.  If-  this 
has  happeded,  chances  are  the  evaluation  won't  help  matters." 

LT  "The  junior  officer  who  is  concerned  enougli  to  care  wliat  his  sub- 
ordinates think  of  him  does  not  need  such  a device." 

LCDR  "Any  .10  who  does  not  get  sufficient  feedback  in  this  area  is  not 
worth  liis  salt  anyway.  Sounds  like  just  another  poor  attempt 
at  leadership  by  the  numbers." 

Peitiaps  the  major  theme  tliat  can  be  gleaned  from  examining  the  comments 
in  Table  111,  is  that  officers  opposed  to  subordinate  ratings  perceive 
favorable  subordinate  ratings  as  resulting  from  the  officer  conducting  him- 
self as  one  super- nice  guy  who  is  lenient  and  lax  with  his  subordinates, 
in  tlie  eyes  of  many  of  the  officers  opposed  to  such  a rating  program,  a 
subordinate  rating  system  would  promote  a popularity  contest. 

Without  arguing  tlie  relative  merits  and  demerits  of  leader  popularity 
per  se , one  can  turn  to  tlie  issue  of  how  do  subordinates,  in  fact,  develop 
their  opinions  of  their  seniors.  Tliat  is,  what  behavior  of  officers  are 
are  deemed  effective  and  ineffective  by  their  subordinates?  The  authors 
and  a number  of  their  students  have  conducted  two  studies  in  attempts  to 
understand  what  leads  an  enlisted  man  to  label  an  officer  as  "effective" 
or  as  "ineffective". 
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Two  Critical  Incident  Studies  Concerning 
Subordinates'  Views  of  Officer  Effectiveness 


Both  of  the  studies  to  be  discussed  used  the  critical  incident  method- 
ology (Flanagan,  195A)  to  gather  responses  from  enlisted  men  concerning 
effective  and  ineffective  officer  performance. 

The  first  study  used  enlisted  personnel  assigned  to  two  Navy  Jet  Air- 
craft Attack  Squadrons.  t)ne  hundred  six  men  representing  pay  grades  E-2 
through  E-9  were  included  in  the  sample.  The  questionnaire  form  asked 
the  respondent  to  describe  an  officer's  behavior  In  the  situation  during 
your  Navy  experience  when  you  considered  jm  action  on  the  part  of  tite 
officer  for  whom  you  worked  to  be  tlie  best  example  of  an  effect ive  Naval 
officer."  Essentially,  Lite  same  question  was  then  later  presented,  asking 
the  respondent  to  describe  an  inef fet:t ive  Naval  officer.  The  reader  should 
notice  that  the  questions  address  effectiveness  and  ineffectiveness  in 
rather  broad  terms,  and  foci  such  as  unit  goals  or  mission  accomplishment 
were  not  provided,  ihese  questions  to  the  respondents  should  not  have 
served  to  preclude  responses  concerning,  say,  discipline  by  the  officer, 
or  the  lack  of  discipline  by  the  officer. 

Table  IV  presents  a summary  of  ilte  critical  incident  categories  formed 
from  the  responses  to  the  two  questions.  (The  original  content  analysis 
yielded  a number  of  more  specific  categories.  These  specific  categories 
were  then  combined  into  the  behaviorally  richer  categories  shown  in  Table  IV.) 

TABLE  IV 

broad  Categories  of  Responses  M.ide 
(Ruu'.erning  Effective  and  Ineffective 
Naval  Officers  - Study  1 
(N=10b  Navy  Enlisted  Men) 


Ef  feet ive 
Naval  Officer 


Inef  feet ive 
Naval  Officer 


Sensitivity  to 
Human  Needs 


-3A 


0 


Professional 

Competence 


-29 


Effective 
Naval  Officer 


Ineffective 
Naval  Officer 


Trust  in 

9 Subordinates  13 

Involvement 

5 in  Job 

Personal 

Characteristics  7 

18  No  Response  23 


106  respondents 


106  respondents 


To  give  the  reader  a flavor  of  the  incidents  obtained  and  categorized, 
the  following  paragraphs  list  some  excerpts  of  representative  incidents. 

Sensitivity  to  Human  Needs,  Effective: 

An  enlisted  man  was  having  problems  with  drugs;  this 
CO  set  up  a drug  abuse  program  which  really  helped 
this  man  and  some  others. 

Sensitivity  to  Human  Needs,  Ineffective: 

After  a plane  crasli  in  which  the  pilot  was  killed, 
this  officer  commented,  "men  can  be  replaced". 

Professional  Competence,  Effective: 

Ship  was  anchored  in  Hong  Kong  when  action  had  to  be 
taken  due  to  a typlioon.  The  LCDR  took  charge  and 
secured  the  ship  while  half  the  personnel  were  off 
on  liberty. 

Professional  Competence,  Ineffective: 

One  of  tile  pilots  didn't  put  his  hands  up  so  we  could 
de-arm  his  guns.  He  by-passed  the  de-arm  area  and 
pulled  his  plane  into  the  line  to  stop  with  his  guns 
pointing  at  the  ready  service  magazine. 
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Trust  ia  Subordinate,  Effective: 


The  department  head  gave  me  the  assignment  and  left  its 
completion  up  to  me. 

Trust  in  Subordinate,  Ineffective: 

My  division  officer  caused  me  to  lose  a launcli  by  in- 
sisting tliat  the  gyros  on  an  air-to-air  missile  wouldn't 
be  uncaged  when  power  was  applied  to  the  plane,  even 
though  all  of  us  in  the  shop  were  experienced  and  told 
him  he  was  wrong.  He  let  us  know  he  didn't  feel  he 
could  believe  us. 

involvement  in  Job,  Effective: 

During  carrier  qualifications,  pilots  in  three  planes 
in  a row  held  brakes  while  going  off  the  catapaults. 

The  division  officer  saw  we  needed  help  fixing  the  tires 
so  lie  pitched  in  to  get  the  job  done. 

i’ersonal  Characteristics,  Ineffective: 

in  my  last  squadron,  my  CXI  was  a heavy  drinker  and  would 
carry-on  about  things.  Tills  negatively  influenced  the 
morale  of  the  men. 

The  autiiors  of  tliis  paper  and  the  three  Navy  officer/students  who 
worked  on  tills  study  noticed  one  other  thing  about  the  incidents  that 
had  been  gathered!  none  of  them  seem  petty  and  whiny.  Most  of  the 
incidents  secmaC.  to  bore  in  on  beliaviors  tiiat  the  Navy  would  want  to 
reward  or  extinguisii  in  its  officer  corps.  Further,  the  sort  of  content 
areas  derived  from  tlie  incidents  look  like  facets  of  performance  the  Navy 
may  wisli  to  consider  wlien  evaluating  officer  performance. 

While  the  researcli  just  described  was  being  conducted,  several  other 
of ficer/students  collaborated  with  the  present  autiiors  in  another  critical 
incident  study.  This  second  study  used  a questionnaire  which  was  answered 
by  289  Navy  enlisted  men  in  pay  grades  E-6  through  E-9,  who  were  at  five 
Fleet  Training  and  tliree  Air  Technical  Training  Centers.  The  questions 
asking  for  critical  incidents  were  somewhat  different  from  tiie  ones  in  the 
first  study,  as  tliey  explicitly  mentioned  mission  accompilsliment . The 
question  concerning  an  effective  critical  incident  asked:  "Think  of  a 

specific  time  (date  and  place)  when  an  officer,  whom  you  worked  directly  for, 
did  something  that  contributed  directly  to  tlie  successful  accomplishment 
of  your  units  mission.  Exactly  what  did  this  person  do  that  was  helpful 
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to  you  or  other  persons  in  the  branch/division/department?"  The  critical 
incident  question  addressing  ineffective  performance  was  similar  to  the 
one  above,  but  asked  about  something  that  an  officer  did  that  actually 
delayed  or  hindered  accomplishment  of  the  units'  mission. 

The  content  analysis  of  the  269  critical  incidents  received  in  this 
second  study  was  conducted  by  a group  of  of f icer/students  working  inde- 
pendently from  the  group  who  content  analyzed  the  data  from  the  first 
critical  incident  study.  The  results  of  the  second  critical  incident 
study  are  summarized  in  Table  V.  (Where  it  would  not  do  violence  to  the 
categories  found  in  the  second  study,  their  labels  were  changed  to  match 
those  found  in  the  first  study.) 


TABLE  V 

Broad  Categories  of  Responses  Made  Concerning 
Effective  and  Ineffective  Naval  Officer 
Performance  - Study  II 
(N=289  Navy  Enlisted  Men) 

Percent  of  the  Total 

Category  Responses  in  this  Category 


1. 

Trust  in  Subordinates 

26.8 

2. 

Professional  Competence 

2A.3 

J. 

involvement  in  Job 

13. A 

A. 

Sensitivity  to  Human  Needs 

11. A 

5. 

Training  of  Subordinates 

10. A 

b. 

Communicat ions 

6.5 

7. 

Safety 

5.0 

8. 

Discipline 

1.5 

9. 

Over  Familiarity  with  Subordinates 

0.7 

A comparison  of  Tables  IV  and  V shows  that  there  is  a high  overlap 
between  the  categories  derived  from  the  critical  incidents  gathered  during 
the  two  studies.  The  second  study  (Table  V)  yielded  five  categories  not 
found  in  the  first  study  (Table  IV):  Training  of  Subordinates,  Communications, 
Safety,  Discipline,  and  Familiarity  with  Subordinates.  The  first  study,  on 
the  other  hand,  yielded  a Personal  Characteristics  category  not  found  in  the 
second  study.  The  authors  suspect  that  tlie  difference  between  the  questions 
in  the  two  surveys  — the  second  addressing  mission  success  while  the  first 
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did  not  — was  the  major  reason  the  categories  found  from  the  two  studies 
did  not  map  even  better  onto  one  another. 

Two  categories  in  iaijle  V probably  require  some  additional  comments, 
as  they  might  seem  to  substantiate  some  of  the  reservations  held  concerning 
subordinate  ratings;  these  are  the  categories  labeled.  Discipline, and  Over 
Familiarity  with  Subordinates.  The  incidents  under  Discipline  typically 
mentioned  punishment  of  a group.  For  Instance,  "Punishment  was  given  to 
the  whole  radio  gang  by  rescinding  special  liberty  when  one  individual  was 
at  fault  for  not  delivering  a message,"  was  one  of  the  Incidents  in  this 
category.  Incidents  in  tlie  category  labeled  Over  Familiarity  with  Sub- 
ordinates, all  referred  to  cases  in  which  junior  officers  bypassed  senior 
enlisted  men  in  the  chain  of  command  when  dealing  with  enlisted  men.  The 
individual  reporting  such  an  incident  apparently  felt  the  chain  of  command 
was  bypassed  because  the  junior  officer  was  too  familiar  with  some  of  the 
junior  enlisted  men,  and  that  this  hindered  mission  accomplishment. 

As  was  the  case  of  the  first  study,  none  of  the  critical  incidents 
gathered  during  tlie  second  study  seemed  of  a petty  nature.  Instead,  they 
usually  addressed  performances  that  the  Aavy  would  cither  wish  to  reward 
or  extinguish.  Neither  study  provided  support  for  the  notion  that  sub- 
ordinate ratings  would  be  greatly  influenced  by  officers  adopting  strategies 
of  ingratiation  in  dealing  with  their  subordinates. 

Discussion 

The  use  of  subordinate  ratings  has  been  discussed  for  many,  irfany 
years.  Based  on  armchair  analyses  of  the  topic,  it  has  been  pointed 
out  that  subordinates  are  in  a position  to  observe  some  of  the  perfor- 
mance of  their  supervisors.  In  many  cases  the  subordinate  is  in  a unique 
position  which  affords  him  opportunity  to  observe  supervisory  performance 
that  is  inaccessible  to  others.  Assuming  that  more  Information  available 
concerning  various  aspects  of  a supervisors  performance  will  permit  a 
better  evaluation  of  liis  over  all  performance,  it  follows  that  subordinate 
ratings  sliould  be  used.  But  even  though  this  was  all  pointed  out  long 
ago,  subordinate  ratings  have  not  been  utilized  in  practice.  This  non-use 
is  evidently  based  on  the  conceptions  of  a "popularity  game",  limited 
perspective,  etc. 

The  most  significant  finding  of  our  critical  incident  studies  in 
this  area,  is  that  the  bases  used  by  subordinates  for  their  evaluations 
of  their  superiors  were  not  consistent  with  such  conceptions.  They  were.' 
for  the  most  part,  based  on  aspects  on  which  most  supervisors  would  want 
to  be  rated.  Tlie  naval  officers,  who  as  part  of  their  academic  training, 
gathered  the  critical  incident  information  for  us  were  originally  sus- 
picious of,  fearful  of,  and  generally  opposed  to,  subordinate  ratings 
prior  to  their  work  in  this  area.  Their  switch  in  attitude  after  having 
gathered  and  analysed  the  critical  incident  information  was  dramatic. 
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Subordinate  ratings,  of  a sort,  have  recently  received  an  upsurge 
in  use  in  academia.  Students'  evaluations  of  teachers  or  professors 
are  no  longer  an  uncommon  practice  and  at  least  one  firm,  the  Apex 
Corporation,  uses  subordinate  ratings.  The  main  resistance  to  their  use 
seems  to  be  based  on  possible  misconceptions  about  what  the  subordinates 
would  actually  be  rating.  If  this  is  so,  what  Information  could  be 
generated  to  correct  these  possible  misconceptions?  Would  it  be  necessary 
to  have  all  supervisors  do  studies  in  which  they  gathered  critical  incidents? 
With  our  recent  renewed  attention  to  the  "people"  aspect  of  organizations, 
subordinate  ratings  may  be  more  important  and  possible  now  than  ever  be- 
fore. Since  our  critical  incident  studies  argue  strongly  against  the 
reasons  usually  given  for  not  having  subordinate  ratings,  the  question 
should  again  be  asked:  Subordinate  Ratings:  Wliy  Not? 
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Tlio  Koeslor  Study  - Electronic  Technicians  Four  Year 
Evaluation  of  Tliree  Types  ol  Training 

Virginia  Zachert,  Ph.D.  Consultant 
Medical  College  of  Georgia 
Augusta,  Georgia 

(For  the  1973  Military  Testing  Association  Conference 
San  Antonio,  Texas,  31  October  1973) 


INTRODUCTION 

A Department  of  Defense  servici’  test  program  was  establislied  in  19b5  at  Keesler 
Air  Force  Base,  Mississippi,  to  explore  ways  to  reduce  the  time  and  cost  of  training 
non-prior  service  airmen  to  perform  as  electronic  equipment  repairmen.  Two  occupa- 
tional specialists  ,ind  tw > markedly  different  approaches  to  training  were  compared 
with  a control  group.  During  the  year  1 96b,  441  graduates  of  the  four  experimental 
courses  were  matched  with  294  control  graduates  of  corresponding  regular  courses  and 
assigned  to  field  units  in  four  commands.  Job  performance  data  were  collected  through 
out  their  first  active  duty  tour  and  an.ilyzed  to  compare  their  success  on  the  job. 


MAJOR  EVALUATION  STEPS 


There  were  fivi'  major  evaluation  steps  .as  follows; 

Alter-Course  Proficiency  Test  at  Keesler  Techinical  Training 
Center  the  first  week  following  graduation.  (Appendix  A) 

First-Job  Proficiency  Evaluation  - first  ten  weeks  on  the 
job.  (Appendix  B) 

Delayed  Field  Evaluation  - nine  months  after  completion  of 
resident  training.  (Appendix  C) 

Second-Job  Proficiency  Evaluation  - after  approximately  two 
years  on  the  job.  (Append i.x  D) 

First  Enlistment  Termination  Survey  - at  completion  of 
initial  active  duty  tour.  (Appendix  E) 


TECHNIQUES  USED 

The  following  techniques  were  used  to  gather  cvalviation  d.ata; 

Performance  and  written  tests  administered  to  graduates. 

Job  Proficiency  appraisals  by  field  supervisors. 

Mailed  questionnaires  completed  by  graduates  and  their  supervisors. 
Field  visits  by  evaluation  team  members. 

Survey  of  records  at  Air  Reserve  Personnel  Center. 
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FINDINGS 


!•  After  over  one  yo.'ir  of  graduate  field  experience,  findings  indicated  that: 

According  to  graduates  and  their  supervisors,  training  provided  in 
the  control  (regular)  courses  is  very  closely  aligned  with  the  needs  of 
field  units  and  is  compatible  with  established  unit  OJT  programs,  career 
development  courses,  and  specialty  knowledge  testing  requirements.  Field 
supervisors  almost  unanimously  oppose  any  reduction  in  the  current  scope 
and  quality  of  resident  technical  training. 

The  shorter  service  test  courses  with  drastically  reduced  electronics 
fundamentals  training  ("X"  course),  while  not  completely  unsuccessful,  re- 
sulted in  several  important  training  deficiencies  which  contributed  to 
initial  and  lingering  graduate  job  difficulties.  Most  apparent  among 
these  were; 

Too  limited  a technical  vocabulary  resulting  in  faulty 
communica  t ions . 

Inability  to  apply,  to  job  needs,  an  understanding  of 
elementary  electronic  phenomena  such  as  capacity,  bias, 
resonance,  transients,  induct.ince,  ets. 

Inability  to  interpret  detailed  circuit  diagrams. 

Insufficient  knowledge  of  the  basic  functioning  of 
classic  electronic  circuits. 

Too  limited  an  ability  to  troubleshoot  (theoretically  and 
practically)  electronic  equipment  at  tlu-  circuit  component 
level.  Field  supervisors  had  to  give  compulsory,  compensatory 
electronic  fundami'nt.i  1 s training  to  graduates  of  thesi'  abbreviated 
training  progr.ims.  Graduates  weri'  very  vocal  in  their  adverse 
criticism  of  tlu’  "X"  type  experiment.il  training  they  received. 

Ihe  shorter  service  test  courses,  which  emphasized  electronic 
f undamint.i  1 s tr. lining  but  de -emphas  i zed  specific  equipmi.'nt  train- 
ing, ("Y"  courses)  were  more  successful  than  the  "X"  courses. 

However,  the  graduates  did  encounti'r  substantial  difficulties 
among  these  deficiencies  in  gradu.ite  hob  proficiency  were: 

Lack  of  sufficient  initial  skill  in  using  published 
technical  data  (technic.il  orders,  m.iintenance  manuals,  etc.) 

Too  little  versatility  and  skill  in  usin;.  standard 
test  equipnii'iit. 

Too  limited  a knowledge  of  electronic  systems 
oper.ition  ,ind  maintenance  techniques  (this  was  very 
pronounced  lor  aircraft  equipment  repairmen  because 
they  were  responsible  for  maintaining  several  different 
types  of  equipm«.nt  I’ven  during  initial  job  assignments). 
Gr.idu.ites  wi  ri  ,ilso  verv  cri.Lic.il  of  this  training. 

Evidence  to  d.ite,  .il  though  Incomplete,  indicates  that  if  field 
conditions  remain  the  same  adoption  of  the  shorter  courses  would: 

Reduce  initial  job  proficiency. 

Reflect  aiiditional  training  workload  and  loss  of 
productive  work  into  operational  units. 

Under  present  standards,  increase  SKT  failures. 

Ri  quire  more  time  on  the  job  to  upgrade  to  the 
"Y"  skill  lev.  I . 

liu  reasi'  the  perentage  ol  gr.idu.ites  believing 
till  ir  Lr.iining  to  have  been  in.idequ.ite  from  3-11  per- 
cent to  33-nO  percent. 
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II.  After  approximately  three  years  of  the  Service  Tost  Program,  findings  indicated 
that : 


The  experimental  graduates  compared  favorably  with  the  control  graduates 
after  three  years  of  job  experience  and  considerable  formal  training  and 
job  oriented  training. 

Experimental  graduates  and  supervisors  were  still  critical  of  the  abbre- 
viated type  resident  technical  training  received  by  the  experimental 
graduates . 

If  field  conditions  remain  the  same,  adoption  of  the  shorter  courses 
would  apparently  be  the  same  as  stated  earlier. 


INFERENCES  THAT  CAN  BE  MADE 


The  service  test  was  conducted  essentially  as  planned;  however,  there  were  some 
difficulties  occasioned  by  the  following  unexpected  developments;  The  intensifica- 
tion of  the  Southeastirn  Asia  Conflict,  several  successive  early-out  programs 
affecting  the  experiment.il  airmen,  and  the  requirement  for  an  ear  1 ier - than-expec ted 
completion  of  the  service  test. 

Considerably  more  data  were  collected  than  could  be  reduced  and  analyzed  with 
the  time  and  resources  available.  However,  all  data  have  been  carefully  recorded 
in  ten  main  report  volumes.  Additional  raw  data  have  been  preserved  in  the  forms 
of  completed  questionnaires,  data  reduction  worksheets,  recorded  comments  by  grad- 
uates and  their  supervisors,  etc. 

Training  received  by  the  control  students  was  very  closely  aligned  to  the  needs 
of  field  units  in  terms  of  graduate  job  proficiency  and  career  development. 

The  job  proficiency  and  career  development  of  graduates  of  both  concepts  of 
experimental  training  fell  short  of  the  needs  of  field  units;  however,  the  applied 
electronic  ("Y")  graduates  were  the  more  satisfactory. 

Both  categories  of  experimental  graduates  required  substantially  more  on-the- 
job  training  and  supervisory  attention  in  the  operational  units  than  did  the  control 
graduates.  Recorded  comments  of  field  supervisors  and  graduates  were  consistently 
critical  of  the  training  deficiencies  in  the  experimental  courses.  The  criticism 
was  especially  strong  for  the  "X”  concept  of  training. 

After  .ibout  threi'  years  into  their  first  active  duty  tour,  847o  of  the  service 
test  graduate  indicati'd  their  intent  not  to  extend  their  active  duty  commitment. 

This  was  gener.illy  borne  out  by  a terminal  survey  of  available  records  in  the  Air 
Reserve  Personnel  Center  (ARPC)  which  showed  that  66%  of  the  graduates  had  in  fact 
terminated  their  first  active  duty  tour  in  four  years  or  less. 

Research  Problem  .\reas: 

1.  Select  sample  for  diads  and  triads. 

2.  Assign  of  control  and  experiment  graduates. 

3.  Maintaining  diads  and  triads. 

4.  Getting  questionnaire  data. 

5.  Analyzing  dati. 

6.  Presenting  findings. 
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Appendix  A 

Sample  of  After  Course  proficiency  Test  (Written) 


WKITTt:N  - prHFORMANCE  (W-JS-l) 
AN/APX-J^ 


1.  A bt*m  h iheik  uf  a k^Y-^^'^A/Al'X-J  ' rcvf.il.Mj  that  rath  MODE 
3,  EMERGENCY  n'ply  iv>nsi>tcfi  uf  tw<>  an<l  'mr-l’all  C'>di* 
trains.  Voltajje  arul  rosistaiur  • hoik>*  ri-vr.ilrrl  Relay 
K- ^^as  upeii. 

A beni  h theck  uf  an  R T- «i7'* / A 1 'X  r<’Vf‘»lf(l  that  itie  AOC/AV'C 
Tifbt  Point  (X-J0‘^,  Pill  \ or  Test  P^iint  T)  tin-asured  • 100  VDC 

Further  t roublestiouting  revirah-d  Eleitron  Tube  V'- 

was  dofri  tive, 

3,  A bench  iheik  of  an  RT-*?9A/APX  revealed  -^SO  DC  at  Test 

Point  Y-l  and  -4h  VDC  at  Test  Point  Y-4.  Resistance  . hecks 
reveali*d  Potentiometer  R- was  open. 

4,  A benih  check  of  an  RT-.J70A/APX  r»’veale<i  Piat  the  trans- 

mitter output  pulses  were  1.7  microse<  onds  in  duration. 
'•Vaveforrn  checks  revcale«i  that  Delay  1-ine  / - was 

defec  tivc*. 

A ben.  h « hi  . k ot  a K Y - A / A PX  - J revealed  that  the  Delay 
line  .'-IrOlA  \cas  oscillating,  thus  produi  me  random  lOde 
pulses  out.  ’A’aveform  »he»ks  revealed  a full  sinewave  input 
t - tt.e  Delay  l-ine  at  Test  Point  M (ORN  1).  Rosisiame  vhe^ks 
reveab-d  tiiat  Crystal  Diode  CH - \cas  open. 

f'.  When  Crystal  f'H-40l,  ir  (he  RT- .17'‘ A/ APX , burns  out  and 

IS  repla«  ed,  t.le.tron  Tube  V- should  als"  be 

replac  ed  as  a preventive  maintenance  step. 

7.  A bench  check  of  an  RT-J7«A/APX  rev.-aled  th.at  the  MODE  2 
and  l/P  replies  both  consisted  of  three  (3)  puls»*s,  1 f.»  inii  ro- 

seconds  apart.  Capa,  itr.r  ( - shoiihl  be  adiusted 

to  .orrect  this  malfumtion. 

A bem  h chei  k of  an  RT--7*^A/ APX  revealed  that  MODE  3 was 
inoperative*.  All  other  MODI.S  are  fun.  tioning  properly.  A 
voltajjc'  chev  k at  Pin  ^ .'f  El«*i  trim  TtiOe  V-  lOh  revealed  0 
{.'F)RO)  volts.  The  probable  cause*  of  this  malfunction  is 
Switch  S-  op«*n. 

'h  fUower  motor  noise  is  ititerro^ating  .m  RT-J79A/APX, 

causing  spurious  repli«*s.  The  first  stajje  to  be  checked  in 
order  to  isolate  this  malfunction  is  V- , 


10.  A ben.  h i heck  of  a K Y A/ APX revealed  output  pulses 
0..i  mif rosecjnds  induration.  Troubleshooting  rev<*aied 
Dela\  I ine  was  defective. 


Appendix  C 

Sample  of  Delayed  Field  Evaluation  Questionnaire  (Graduate) 


N-X-137  N-C=137 


3.  ON-THE-JOB  TRAINING: 


a.  Do  you  feel  that  technical  trair.mu  at  Keesler  adequately  nreqarecl  vou  lor  par- 
ticipation in  your  unit's  f Df  program'’  □ yes,  □ no.  If  your  answer  is  no. 
explain.* 


Yes-X  (-  1 2J 


No-X  rt2;  C \A 


b.  Did  the  technical  tr. lining  you  rec»  iveu  ,it  Keesler  provide  \'ou  with  an  aflequal* 
foundation  to  further  prepare  on-the-jeb  for  tin  .SKT'’  □ yes,  O no.  If  vour 
answer  is  no,  explain. 


Yes-X  47;  f 125 

Yes-X  83;  C-80 


.\o-X-.8d;  r -12 
.-.o-X.  5i;r  St'i 


Om i.s.s  lo.ns 


nils  ..  io"  s 


c.  Have  vou  tak.  n the  "5"  )e\ .d  .SI.  I . □ . □ no.  If  ve-,,  -.ni  von 

encounter  anv  particular  diffu  nltv'’  Dyes,  n no.  Ifves,  expl.un.' 


Yes-X=h7;  C 3’ 


' > u - X . i S ; l 1 


Onii lon.s  - 


d.  \ on  It  el  ..lat  von  .'i..ve  ij.  t r.  il»  . u>  u.im  stitisfa.  ter’.'  p : of  ic  ler.  e \ ii.  \-on  r 

norniai  j.jh  r.iu'i.-ies  withni  .■  riasonahli  :te-ui;  . nr  - ioi.>  ' .Vi  niine ’’  d]  ye- 

□ no.  If  no,  ex  ila  in  . 


■'<■.(  '21 


O'  II  . 


e.  Wn.it  .1  pe  in.iiuli.  aiice  wonld  von  pr ' I'er  to  pe  fortn  m .o,ir  unit 
Check  E oni  block  to  indii.ile  vour  '"i  st  . hoice. 

d]  Flight  line  X 21  f -Is 

E r ombin.  f ilig'il  1.1,.  ,i;ul  si. on  .X  1(h)  f ' 1 
dl  I’f  rioiin  uoi  1 X '■  C-  a 
□ No  pn  ferenct  X . i 3 
C Other  (cxjilain)  Shop  X 13  f-23 


n X 


Appendix  B 

Sample  of  Delayed  Field  Evaluation  Questionnaire  (Graduate) 


H»»  ofcility  to  ap»ly  »Mcc»»»«wMy,  wwintwmic*  pnwtiyUi 

«•  NOT  IMPORTANT:  This  it  « **nie«  to  know"  tybjoct  oroo  toMom,  if  ovor,  uto4  by  mo  i*  my  cufront  (oh.f 

b.  somewhat  IMPORTANT:  My  undortton^inf  of  ond  ability  to  opply  mointononeo  ^rineiptot  in  thu 

tubioct  oroo  Qro  opplicoblo  to  only  o tmoll  portion  of  my  cyrront  jobA—'  ■■  ■ - , — - — , ^ 

c.  HIGHLY  IMPORTANT:  I theyld  undorttond  ond  bo  oblo  to  opply  principlot  in  thit  tubjoct  oroo 

boeoyto  it  opphot  to  o mojor  portion  of  my  cyrront  job  iti - - - — 

d.  CRITICALLY  IMPORTANT:  My  undorttondinp  of  ond  obility  to  opply  mointononeo 

principlot  in  thit  tybjoct  oroo  oro  inditpontoblo  ropordlott  of  tho  fropyoncy  of  itt 
oppliCQtion  in  my  curront  j^^  ^ — _ _ _ _ _ _ 

2.  ADEOLTACY  of  TRAINING:  Indicoto  if  troining  ^^j^nFQlfATE  i 
rocoivod  ot  Kootlor  wot  inodopyoto,  odopyoto  or  nne  tuau 

^ ^<f*qo_qto. - i ADEQUATE  * 1 

SUBJECT  AREAS  H **  A 1 CRIT  HIGH  SOME  N01 

DC  Fundamentals  1^1  ^ 34  103  35  63  34  3 

Icl  10  124  2 53  35  2 

AC  Fundamentals  p|  ° '1°  38  63  32  1 

Icl  6 121  6 2L-  __i 

Vacuum  Tubes  p|  ° ^0  77  27  75  30  1 

Icl  8 113  8 32  75  22  1 

Solid  State  Devices  |^|  ^ 43  53  12 

Icl  3 

Basic  Electronic  Circuits  I I ° 19  118  b3  64  6 1 

Icl  7 1 15  13  57  51  24  0 

RF  Generation  and  Transmission  1^1  39  3 

Icl  3 111  21  35  67  _28 ^ 

Receivers  ■*  |^|  ^ 32  6')  67  8 0 

Icl  8 121  6 22_  _2 0_ 

Transmitters  p|  ° ^3  : TT'  9 0 

Icl  5 114  15  69  _59 6 0_ 

Timers  p|  ° 41  95  20  48  |44  22 

Icl  6 89  40  18  46  [43  26 

Indicators  |^|  * ^4  61  24  il  3i>  2 

Icl  4 93  39  26  ’.4  36 1_ 

Power  Distribution  and  Control  Circuits  1^1  ^ 30  74  |j-  4 1 34  i 

Icl  1 113  2l|:.4  I 62  |<1  5 

Test  Equipment  P |34  101  I 2 1 ■ t-i  0 

Icl  4 85  4 7 |6><  4b  l_ 

Troubleshooting  Logic  1 1*^  1 7 83  41  4 0 

Icl  4 87  45  95  3o  _3 0_ 

System  Fault  Isolation  P 26  86  44  2 0 

Icl  2 38  45  83  « 5 1_ 

Electronic  Circuit  Tuning  and  Alignment  P 42  b2  56  15  1 

Icl  3 97  36  59  bl  J_2 0_ 

Repair  Techniques  P|  ^ 31  <3  4 () 

IC  I 3 87  46  51  2 L 

Use  of  Handtools  Pf ^ 6 3 1 71  26  1 

let  2 98  26  28  74  28 3_ 

Use  of  Maintenance  Publications  ^ ^ 

ICIO  53  83  58  61  1_5 0_ 

Spare  Parts  Acquisition  Pi  ' 54  15  79  36  4 

IcIO  63  73  25  63  39 

Other  All  omissions  not  shown  in  totals.  I I 


X 

0 

34 

103 

35 

63 

34 

c 

10 

124 

> 

43 

53 

35 

X 

0 

27 

110 

38 

63 

32 

c 

6 

121 

42 

57 

32 

X 

0 

60 

77 

27 

75 

30 

c 

8 

1 IS 

8 

32 

75 

22 

X 

0 

1 1 

1 26 

26 

43 

53 

c 

3 

be 

Jl. 

X 

0 

19 

118 

b3 

64 

6 

c 

7 

115 

13 

57 

51 

24 

X 

0 

44 

91 

32 

59 

39 

c 

3 

111 

21 

35 

67 

28 

X 

1 

84 

52 

50 

67 

8 

c 

8 

121 

6 

70 

57 

7 

X 

0 

81 

bi 

9 

c 

5 

114 

15 

b9 

59 

6 

X 

0 

41 

95 

20 

48 

44 

c 

6 

89 

40 

18 

46 

43 

I 

74 

hi 

J4 

71 

36 

c 

4 

93 

39 

■.4 

3 b 

X 

2 

60 

74 

34 

c 

1 

113 

2 1 

34 

62 

f 

X 

24 

101 

1 2 

5i. 

£ 

4 

85 

47 

4b 

y 

X 

19 

101 

17 

HH 

41 

4 

C 

4 

87 

45 

95 

3o 

3 

X 

20 

91 

26 

86 

44 

"> 

i. 

c 

2 

88 

45 

83 

43 

5 

5c 

13 

81 

b2 

56 

15 

c 

3 

97 

36 

59 

bl 

12 

X 

7 

98 

31 

■i" 

73 

c. 

3 

87 

46 

51 

X 

27 

102 

6 

3 1 

7T 

26 

c 

12 

?8 

iib 

28 

74 

28 

X 

13 

83 

1 1 

46 

65 

12 

c 

0 

53 

83 

58 

61 

15 

X 

5 

77 

~54 

IS 

79 

36 

c 

0 

63 

73 

2.S 

63 

39 

241 


Appendix  C 

Sample  of  Delayed  Field  Evaluation  (Supervisor) 


N = X-141  N = C-142 


I.  HOW  ESSENTIAL  hit  (ob  is  ths  fro^uots't  procticof  knowlsdg*  of,  and  obiLty  to  apply  tuccottlu 


o.  NOT  IMPORTANT:  This  is  a **nica  to  know**  subjoct  oroo  soidom,  if  ovor,  ro^uitod  by  tho  yradwoto  m 
hit  cuffoni  job.*  ■ I ■■  ■ I I — ■ ■■■  -I  ■ — ■ - I 


b.  SOMEWHAT  IMPORTANT:  Groduoto  should  undorttond  ond  bo  oblo  to  opply  principlot  m tl 
subjoct  ofoo  but  it  It  only  opplicoblo  to  o ttnoll  portion  of  hit  curront  job  osupnmont.o  — 

c.  HIGHLY  IMPORTANT:  Gtoduoto  should  undorstond  ond  bo  oblo  to  opply  pfinoiplot  in  tho 
Sub|oet  oroo  bocouso  it  is  oppltcoblo  to  o mojor  port  of  his  curront  job  ossiprtmont,^—  — 

d.  CRITICALLY  IMPORTANT:  Groduoto*t  undorstonding  and  obility  to  opply  prineiplos  «n 
tho  subjoct  oroo  oro  indisponsiblo  rofordloss  of  tho  fropuoncy  of  its  application  to 

his  cgrront  fiold  dutios.wi^— — "■  ■—  ■ ■ ■ . 


2.  GRADUATE'S  JOB  PROFICIENCY:  Indicoto  tho 
groduoto's  domonstrofod  proficioncy  in  opplying 
tho  prineiplos  of  ooch  Suojoet  oroo  bolow  to  tho 
lirotnonts  of  his  job. 


OUTSTANDING  •— 

SATISFACTORY 

UNSATISFACTORY 


SUBJECT  AREAS 

c 

u* 

o 

ICRIT  HIGH  SOME  NOT  1 

DC  Fundamentals 

X 

C 

35 

3 

96 

102 

7 

33 

63 

68 

65 

58 

1 1 
13 

0 

0 

AC  Fundamentals 

X 

c 

40 

4 

92 

101 

7 

33 

62 

68 

66 

58 

10 

13 

0 

0 

Vacuum  Tubes 

X 

c 

32 

2 

99 

107 

8 

29 

48 

47 

79 

77 

12 

14 

1 

0 

Solid  State  Devices 

X 

c 

66 

30 

68 

95 

5 

13 

4 5 
36 

53 

58 

36 

39 

5 

5 

Basic  Electronic  Circuits 

X 

c 

42 

5 

88 

99 

9 

34 

74 

77 

58 

52 

6 

9 

1 

0 

RF  Generation  and  Transmission 

X 

C 

38 
1 1 

89 

103 

10 

23 

45 

43 

69 

73 

23 

21 

1 

1 

Receivers 

X 

C 

17 

3 

105 

100 

17 

35 

75 

72 

60 

61 

4 

6 

0 

0 

Transmitters 

X 

c 

19 

7 

104 

99 

16 

31 

71 

69 

63 

64 

. 5 
6 

0 

0 

Timers 

X 

c 

38 

13 

87 

106 

6 

13 

34 

32 

57 

53 

31 

41 

15 
1 2 

Indicators 

X 

c 

24 

5 

104 

103 

10 

19 

44 

40 

76 

80 

18 

16 

1 

2 

Power  Distribution  and  Control  Circuits 

X 

c 

25 

6 

103 

106 

10 

26 

56 

56 

69 

69 

1 2 
13 

1 

0 

Test  Equipment 

X 

c 

10 

10 

106 

100 

23 

28 

92 

92 

43 

42 

4 

5 

0 

0 

Troubleshooting  Logic 

X 

c 

23 

17 

87 

80 

29 

41 

104 

102 

34 

37 

1 

0 

0 

0 

System  Fault  Isolation 

X 

c 

21 

13 

94 

81 

23 

43 

96 

92 

40 

43 

2 

3 

0 

0 

Electronic  Circuit  Tuning  and  Alignment 

X 

c 

21 

8 

96 

96 

22 

34 

83 

77 

43 

48 

13 

14 

0 

0 

Repair  Techniques 

X 

c 

13 

10 

96 

89 

30 

39 

83 

86 

53 

48 

3 

5 

0 

0 

Use  of  Handtools 

X 

c 

3 

3 

98 

100 

38 

35 

57 

55 

77 

80 

6 

4 

0 

0 

Use  of  Maintenance  Publications 

X 

c 

11 

10 

94 

104 

34 

24 

66 

57 

63 

74 

10 

8 

u 

0 

Spare  Parts  Acquisition 

X 

c 

8 

5 

103 

111 

26 

21 

34 
3 b 

70 

73 

28 

27 

3 

•> 

Other  (Identify)  All  omissions  not  shown  in  totals. 
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Sample  of  Delayed  Field  Evalual 
Trainabtlity  Questionnaire  (I 


•N  - X-1  16  -N  = C-1  M 

1 

1 

I 

b.  Check  the  appropriate  squares  in  the  t£ 
of  the  graduate's  performance  in  the  TECHNIC/ 


X c 

GRAnCATE  TECHNICAL 
TRAINING  PIIRFORMANCE 

1 

Rating  Seale  | 

Far 
Be  low 
A ve  rage 

TRAINING  TASKS 

Interest  and  P^nthusiasm  ! 

0 □ i 

Partie  ipaticm  in  T raining  Class 
Discussions 

n □ . 

Ability  to  Ask  TliougUtiui  i 

Questiijns 

1 n □ 1 

Assistance  and  Counseling  j 

Required 

2 □ 9 

Undf'rstaudinc  of  T<.*chnikal  | 

Termiriology 

1 □ 0 

Understanding  of  Elec  trunie  j 

Principles 

2 □ 0 

Practical  Application  of 
Electronic  Principles 

Practical  Knowledge  of  1 

Basic  Circuits 

[ 0 □ 0 

Application  of  Troubleshooting  | 

Logic 

^ n 2 

Use  and  Interpretation  ot  Test  1 

Equipment  i 

0 □ 0 

Use  and  Interpretation  of  ] 

T ei  iit.il  a i Data  I 

0 □ 0 

Overall  1 "-ai  na  bi  1 ■ i y j 

0 □ 0 

All  training  tasks  were  not  rated  oy  each 


2 


SampU'  ol 


Appi‘n<Jix  D 

S.  i ond-.lob  Proticicncy  Kvaluation  Questionnaire 


(Graduate) 

N X-12Z  C-123 


8.  In  yottr  Judgraent  which  of  the  following  courses  in  the  DOO/tJS^  Serrlce 
Test  Progran  would  you  recomnd  for  training  futuz*e  radar  repairaan? 

X r 

so  Id?  I I "C"  (Regular  Cobtrol  Course) 

2 7 10  n 'T'  (Cue-Responae  Course) 

15  b Cl  "Y”  (Applied  Electronios  Course) 

Briefly  explain  your  reason  for  selecting  the  abovp  course. 


Conpared  with  other  skill  lewl  repaimsn  wltii  whom  you  hare  worked, 
how  do  you  rate  yourself  on  the  foUcwlng  evaluation  items?  (Check  0 coie 
appropriate  square  for  each  item.) , 


EVALUAnON  ITEMS 

X c; 

X c 

iU 

nn 

X c 

X c. 

X c 

Far  ^low 
Avera^ 

Below 

Average 

Average 

Above 

Average 

Far  Above 
Average 

Amount  of  Useful  Work 
Produced 

0 □(. 

“ □'> 

"O” 

* □ » 

Quality  of  Work  Pro- 
duced 

n □„ 

' □" 

•'Q ' '■ 

isQui 

Speed  in  Coiqpletlng 
Assigned  Maintenance  T^ste 

‘ □' 

18  □ ^ 

bl|  1 bb 

5 ni4 

Atdllty  to  Work  Without 
Close  supervision 

« U" 

"O 

5 s[  1 50 

7 □>! 

t^u^r standing  of  Technical 
T^mdnolo^ 

10  □o 

1 40 

7 

Understanm.ng  of 
Eleotrotxic  Principles 

2 [n  9 

Practical  Application  of 
Electronic  Principles 

1>.  Qo 

57  □'  ^ 

'' 1 1 

, Dio 

)hractical  Knowledge  of 
Basic  Circuits 

2S  ^0 

bo  □r/ 

'Qb2 

'□  '•> 

1 □ « 

Application  of  1)700110- 
shooting  Logic 

1 □() 

uQ  ' 

(.h]  |54 

' 019 

tTse  and  Interpretation 
of  Test  EQulpnsnt 

' □' 

'■  □ ■’ 

I^se  and  Interpretatlan 
of  Technioal  Liata 

i2n  '■ 

7Q77 

< □ .. 

Qeneral  Technical 
Ability 

1 Qo 

« [n  1 |8-[^70 

2«G40 

0 □!> 
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ADpendlx  D 

Sample  of  Second-Job  Proficiency  Fjvaluatlon  (Supoivisor) 
>!'N-X-134  C-135 


1.  SUPERVISOR'S  EVALUATION  OF  GRAIXIATE  JOB  KNOWLED® 
INSTRUCTIONS:  Please  check  each  knowledge  item  listed  below  in  the  appro- 

priate  column  to  Indicate  your  judgment  of  the  graduate's  subject  knowledge 
level.  Base  the  rating  on  your  obeerratlon  of  this  graduate's  performance 
while  under  your  superrtsion. 

SUBJECT 

KNOWLED® 

LEVELS 

Scale 

Value 

DEFINinON 

NA 

Not  applicable  to  or  has  not  demonstrated  his 
knowledge  in  his  current  assignment. 

D 

EVALUATION.  Can  evaluate  conditions  and  make 
decisions  about  the  subject. 

j 

C 

ANALYSIS,  Can  analyze  facts  and  prlnci 
and  draw  conclusions  about 
the  subject. 

pies 

B 

PRINCIPLES.  Can  explain  relation- 
ship  of  basic  facts  and  state 
general  principles  about  the 
subject. 

A 

FACTS.  Can  identify  basic 
facts  and  terms  about  the 
subject . 

TRAINING  STANDARD  ITEM 

XA  C 

X B C 

X C C 

X D C 

XNA  C 

— 

ELECTRONIC  FUNDAMENTALS 

/ / / 

! / / / 

/ / / / 

y ,/ /j 

' / / / 

1.  DC  Circudts 

1 3 0 

Z1  16 

32  41 

46.  6 1 

14  10 

2.  AC  Circuits 

7 

34  19 

27  41 

44  57 

3a  Sclid  State  Devices 

!4 

31  3 1 

34 

18  30 

3 3 25 

U.  ’Vacuum  Tubes 

12  3 

■ 

30  16 

3 2 47 

44  5b 

_L4_i0 

5.  Oscillators 

n 8 

26  24 

4 3 46 

27  45 

1_5  _1I 

6.  Receivers  Principles 

i5  S 

26  22 

41  48 

34  43 

lb  16 

i 7.  Motors  and  Servomechanisms 

1 t.  5 

28  24 

42  44 

26  4j 

- J ! 

1 

8.  Waveshaping  Circuits 

19  11 

26  2 1 

40  5 3 

28  38 

lb  11 

9.  Microwave  Principles 

15  11 

2 5 22 

24  29 

18  25 

50  47 

FJICnON  AND  TACnCAL  USE  OF  ACW  RADAR 
EQUiriniT 

/ / / 
/ n 

/ / / 
L±1_L 

/ / / / 
[///_/ 

^/// 

/ / / 
fjjj. 

1 

10.  Search  Radar 

8 5 

17  9 

2 5 30 

24  28 

58  62 

11.  Height  Finder  Radar 

4 6 

15  10 

25  24 

48  42 

40  52 

12.  Gap  Filler  Radar 

8 G 

7 

5 8 

7 5 

105 

112 
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Appendix  D 

Sample  of  Second-Job  Proficiency 
Evaluation  (Supervisor) 


DEFINITION 

NA 

Not  applicable  to  or  not  perfonwd  by  the  graduate 
in  Ms  current  assicnaent. 

HIGHLY  PROFICIENT.  Ca 

n do  the  complete  task 

D 

quickly  and  acoarately 
how  t.Q  do  the  tank . 

0 Can 

tell  or 

show  others 

TASK 

COMPEIEN?,  Can  do  all 

parts 

of  the  task 

• 

PERPORMINCE 

LEVELS 

3 

Needs  only  a spot  check  or  completed 
work.  Meets  sdninium  local  demands  for 
soeed  and  accuracy. 

PARTIALLY  PROFICIENT. 

Can  do 

most 

2 

parts  of  the  task.  Needs  help  only 
on  hardest  parte.  May  not  meet 
local  demands  for  speed  or  accf’racy. 

EXTREMELY  LIMITED.  Can  do 

1 

simple  parts  cf  the  task. 
Needs  to  bs  told  or  shewn 
feviw  to  do  most  C'f  the  task. 

TRAINING  STANDARD  ITEM 

SBB 

X 2 c 

X 

3 C 

X 

U c 

XNA  c j 

10.  Uses  main 

tenants 

* fcrTna 

_i 

1 1 

12  18 

54 

51 

44 

49 

23 

16 

ADJUSTS  AMD  USES  lEST  EQU.'PMENI  SICH  AS 

■/ft 

/ / / , 

' / / / 

' / / / 

/ / / 1 



11 0 Multimeters 

0 0 

9 0 

42 

42 

71 

84 

12 

9 

12.  Vacuum  tube  voltmeters 

■ 

0 0 

1 1 1 

42 

45 

68 

79 

1 3 

10 

13.  Signal  general  .V 

3 

3 2 

16  9 

60 

63 

40 

53 

1 5 

8 

IL.  OsclUoso 

opes 

4 1 

10  6 

54 

54 

54 

65 

12 

9 

15.  ‘Spectr.un 

analj-zars 

9 5 

27  23 

46 

57 

28 

29 

'i.l 

16 . Power  me tars 

5 4 

23  10 

39 

53 

48 

54 

19 

14 

17.  Crystal  checkers 

6 1 

18  17 

36 

49 

22 

22 

52 

46 

19.  Transistor  testers 

8 2 

19  22 

40 

37 

9 

19 

56 

55 

20.  Frequercy  measuring  davloes 

8 2 

23  19 

49 

48 

27 

48 

27 

18 

21.  MTI  evaluators 

5 

11  11 

23 

1 9 

4 

1 3 

87 

89 

22.  Transponder  test  sets 

6 3 

9 12 

19 

30 

23 

1 5 

77 

_J5 

23.  Selects  the  proper  tast  equipment  and 
repair  tools  in  raalntairlr.g  AGW 
equipment 

' * 

1 5 9 

56 

58 

48 

58 

12 

9 

2li.  Recognizes  defective  test  equipment 
through  indications  obtained  during 
operation 

3 3 

12  15 

56 

49 

4b 

59 

1 5 

9 
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Append ix  E 

Sample  of  First  Enlistment  Survey  (Graduate) 


2.  Date  separated  from  the  Air  Force 

day 

3.  Base  of  separation: 


4.  Military  pay  grade  at  the  lime  of  separation  from  the  Air  Force; 

□ E-l,  DE-2.  DE-3,  QE-4,  DE-5,  □Other 

5.  Marital  status: 

a.Q  Single  b.  □ Married  c.  □ Divorced  d.D  Widower 

6.  Number  of  dependents; 

Ql,  Q3,  0*4,  Other 

7.  Do  you  have  any  plans  at  present  for  returning  to  active  mililarv  duty  O Yes, 

□ No.  If  yes,  please  list  date.  day  month  year. 

8.  Are  you  now  a member  of  an  active  reserve  unit?  DVes,  DNo. 

9.  In  your  judgment,  did  the  technical  training  you  received  at  Keesler  AFB  adequately 
prepare  you  for  satisfactory  career  progression  in  the  Air  Force?  O Yes,  □ No. 

If  no,  please  explain  briefly.  

10.  Please  check  Ef  the  following  statement  concerning  your  employment: 

□ Full  lime,  DPart  lime,  DUnemployed 

1 1.  If  you  are  now  self-employed,  give  a brief  description  of  your  work. 


month 


year. 


12.  If  you  are  now  employed  by  someone  otlior  than  yourself,  please  complete  the  follow- 
ing items: 


a.,D 

Marne  of  employer 

b.D 

Type  husinr.sp 

C.D 

Brief  description  of  work 

13.  If  you  are  employed,  does  your  job  require  any  knowledge  or  skills  on  which  technical 
training  was  proyided  at  Keesler  AFB’  QYos,  ONo.  If  yes,  please  list  and  explain 

briefly . 
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Appendix  E 

Sample  of  First  Enlistment  Termination  Survey  (Supervisor) 


X-83  C-80 


r>K grf.es  of  supervision 


Unsiipprvised:  Can  perform  the  whole  task  by  himself  without 

4 supervision  or  technical  assistanc  e.  Can  tell  cir  show  others  how  to 
perform  the  task. 


^ Occasionally  Supervi seel:  Can  perform  all  parts  of  the  task. 
Req\iires  only  .i  spot  check  of  completed  work. 


Frequently  Supervi  siul:  Requires  supervision /techni- 
2 cal  assist  mce  practicallv'  throughout  the  task.  Can 
perfoj-m  onl\  simple  parts  of  the  task  unsupervised. 

I Continuail-  Supervised:  Requires  supervi- 

sion/t  echni<  a I assistance  throughout  the  task. 

PERFORMANCE  TASKS  1 2 

X C X C 


2.  Uses  of  technical  publications  such  as;  ///|//  / / //  l/  //  / f/  / 


X *C  X^C  x^c  x'^c 


I.  Wiriru^  dia  i;rams 


).  Circuit  dia^trams 


c.  Parts  provi sionim;  and  illustrated 
parts  breakdown 


. Alignment  procedures 


3. 

Troubleshooting  of  electronic 

equ  ipment : 

a. 

laicaU^es  malfunct  ion  to 
assembly 

major 

b. 

TocaIizt‘s  malfunction  to 
sub.i  s sembly 

2 0 


2 0 9 6 


I 

0 I 3 3 20  2 1 i 58  61  i 


4 26  27  44  51 


16  2 1 59  63 


, l.vu  ciUzi  s Ifunrlion  lo  a circuit  ] 2 0 j N 7 | 36  36  j 38  39 


1 on  3 31  31  3S  48 


d.  I .iK  a I i / » s f-n  « 1 f'lnct  ion  a c i rr  nil  • 

< ompont'nt  f'Jurh  as  capacitor, 

r.  -ii'.tor,  ' n 3 0 10  6 40  31  38 

f,  . '•jtiipmrni  to  nnrformancc 

sf.inrlard^  1 0 11  3 31  31  38  48 

4.  I ran.sf*  r <*f  kr'»-.vl«  rh  - skills  from 

fa  mi  1 1.»  r 1 1»  unf  • . i r • irc  t ronit 

rquil'm*  Ml  I 3 I IP  n 38  39  19  26 

NOTE'.:  Ail  ^ ..  • i i »'i  - <hd  n*  r rate  all  ^radnatfS  on  all  items. 


PROCESS  VERSUS  PRODUCT  MEASURES  IN  PERFORMANCE  TESTING 


William  C.  Osborn 

Human  Resources  Research  Organization 

Consider  the  following  situations. 

After  having  undergone  training  in  tank  gunnery  a soldier's  proficiency 
is  being  tested  on  a gunnery  range.  During  the  course  of  this  test  he  will 
fire  several  main  gun  rounds  at  targets  varying  in  size,  shape  and  distance. 
In  each  case  his  score  is  determined  by  whether  he  hit  the  target  within 
some  specified  time  limit,  arid  he  is  certified  a tank  gunner  if  he  scores 
above  some  minimum  level  required  for  qualification. 

Under  other  circumstances,  a soldier  having  undergone  similar  training 
may  be  evaluated  differently.  Let's  assume  that  ammunition  is  scarce  or 
that  adequate  range  facilities  are  not  available.  Here  the  soldier  might 
have  to  be, tested  under  dry-firing  conditions.  He  would  be  required  to 
take  actual  or  miniaturized  versions  of  targets  under  fire,  and  « tester 
would  assess  in  each  case  whether  the  gunner  1)  acquired  the  target  with 
smooth  manipulation  of  the  hand  controller,  2)  correctly  ranged  on  the 
target,  3)  achieved  the  proper  sight  picture,  4)  squeezed  the  firing  switch 
without  losing  the  sight  picture,  and  5)  fired  within  some  allotted  time. 

Here  the  gunner  is  qualified  if  he  performed  each  of  the  five  procedural 
steps  correctly  on  some  minimum  number  of  targets. 

In  the  first  situation  described,  a task  outcome  or  product  measure  — 
target  hits  — is  the  basis  for  evaluating  gunners;  whereas,  in  the  second 
instance  correct  task  procedure  or  a process  measure  is  the  basis  for  evalua- 
tion. Though  somewhat  oversimplified,  the  contrasting  approaches  to 
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performance  testing  drawn  in  these  two  examples  illustrate  the  focus  of  this 
paper:  the  use  of  process  versus  product  measures  in  performance  testing. 

I am  chiefly  interested  in  the  use  of  performance  tests  to  evaluate  the 
results  of  training,  and  to  properly  set  the  stage  for  what  I have  to  say 
today  let  me  first  summarize  what  the  training  evaluator  considers  to  be 
the  ideal  use  of  product  and  process  measures.  Performance  tests  are  used  in 
training  evaluation  to  serve  two  purposes:  (a)  to  certify  student  achieve- 

ment, and  (b)  to  diagnose  weaknesses  in  the  instructional  system.  In  the 
use  of  such  tests,  proficiency  measures  which  focus  on  task  outcomes  (products) 
normally  provide  data  relevant  to  the  first  purpose,  whereas  measures  of  how 
the  tasks  are  carried  out  (process)  pertain  to  the  second.  For  example,  the 
number  of  targets  hit  by  the  tanker  trainee  would  be  the  product  measure  by 
which  his  qualification  as  a tank  guraier  is  assessed.  However,  if  he  fails 
to  qualify  we  would  also  like  to  know  why  — where  was  his  training  weak? 

This  is  where  process  measures  are  useful:  if  the  gunner  consistently 

missed  targets,  was  it  because  he  ranged  incorrectly,  or  obtained  an  improper 
sight  picture,  or  wasn't  able  to  maintain  the  gun  lay  during  firing?  This 
type  of  data  is  useful  in  diagnosing  areas  of  training  deficiency,  and  is 
essential  in  efficiently  remediating  trainees. 

Thus  we  see  the  roles  played  by  product  and  process  measures  in  training 
evaluation.  Both  are  important  — even  critical  — when  used  for  their 
respective  purposes  in  evaluating  the  results  of  training. 

Product  and  Process  as  Measures  of  Student  Achievement 

With  this  background  I would  now  like  to  narrow  the  focus  of  my  comments 
to  the  use  of  these  types  of  performance  measures  In  serving  just  the  first 


250 


of  the  two  evaluation  purposes  stated  above  — that  of  certifying  student 
achievement.  In  testing  a student  to  determine  if  he  is  qualified  to 
advance  to  the  next  level  of  training,  or  ultimately  out  of  training  and  on 
to  the  job,  we  would,  as  mentioned,  normally  prefer  to  use  a product  score. 
Before  a man  is  certified  as  a gunner  we  would  like  to  have  him  demonstrate 
that  he  can  hit  targets;  or  before  certification  as  a navigator  he  should 
actually  demonstrate  that  he  can  get  from  point  A to  point  B;  etc. 

Although  it  may  safely  be  said  that  every  task  has  a purpose  — the 
fact  of  the  matter  is  that  in  practice  a great  many  performance  tests  are 
used  which  employ  process  measures  only  in  evaluating  student  achievement 
or  job  readiness.  hTiy  is  this  the  case?  Is  the  substitution  of  process  for 
product  measurement  justified?  If  so,  when?  If  not,  how  may  the  test 
developer  improve  his  methods?  These  are  the  questions  that  I will  be 
addressing  today. 

Product,  Process  and  Types  of  Tasks 

Before  exploring  in  more  detail  the  issue  of  why  process  measures  are 
so  widely  substituted  for  measures  of  task  product,  it  will  first  be  helpful 
to  consider  three  categories  of  tasks : 

1.  Tasks  in  which  the  product  ^ the  process. 

2.  Tasks  in  which  the  product  always  follows 
from  the  process. 

3.  Tasks  in  \.'hich  the  product  may  follow  from 
the  process. 

Relatively  few  tasks  are  of  the  first  type,  those  in  which  product  and 
process  arc  one  and  the  same.  These  are  normally  tasks  which  serve  an 
aesthetic  purpose  such  as  gymnastic  exercises  or  springboard  diving.  Close 


order  drill  is  a good  military  example.  Here  we  see  that  the  outcome  or 
product  of  the  task  is  no  more  or  less  than  the  correct  execution  of  steps 
in  task  performance  — that  is,  the  process. 

More  tasks  are  of  the  second  type  mentioned,  those  in  which  the  product 
invariably  follows  from  the  process.  Fixed-procedure  tasks  typically  fall 
in  this  category.  Troubleshooting  an  electrical  circuit,  disassembling  a 
rifle,  and  implanting  a land  mine  are  examples.  In  tasks  of  this  type  the 
procedural  steps  are  known,  observable,  and  comprise  the  necessary  and 
sufficient  conditions  for  task  outcome;  so  if  process  is  correctly  executed, 
task  product  necessarily  follows. 

A great  many  job  tasks  are  of  the  third  type  where  the  product  is  less 
than  fully  conditional  on  the  process.  In  other  words,  with  these  types  of 
tasks  the  process  may  appear  to  have  been  correctly  carried  out  but  the  goal 
or  product  not  achieved.  This  can  happen  for  one  of  two  reasons:  either 

(a)  because  we  are  unable  to  fully  specify  the  necessary  and  sufficient  steps 
in  task  performance  or  (b)  because  we  do  not  or  cannot  accurately  measure 
them.  In  aim-firing  a rifle,  for  example,  we  are  interested  in  knowing  if  a 
soldier  is  standing  with  his  body  properly  oriented  to  the  target,  face 
properly  positioned  on  stock,  rifle  sling  in  correct  position,  lead  arm 
perpendicular,  and  firing  arm  parallel  to  ground;  if  he  is  breathing  correctly, 
has  a good  sight  picture,  and  squeezes  the  trigger.  Presumably,  if  this 
process  is  followed  the  rifleman  will  hit  the  target.  Assuming  that  we  have 
identified  all  essential  steps  in  rifle  firing,  and,  further,  that  we  can 
reliably  measure  their  correct  execution,  then  the  task  is  of  the  second 
type  described  above  and  does  not  belong  in  Category  3.  However,  in  practice, 
we  know  that  our  best  efforts  to  evaluate  execution  of  this  particular  task 
are  not  sufficient  to  warrant  substituting  process  for  product  measurement. 


I 
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In  other  words,  sometimes  the  target  is  missed  even  though  in  the  judgment 
of  a skilled  evaluator  the  rifleman  did  everything  right.  Therefore,  either 
because  we  are  not  absolutely  certain  that  we  have  identified  all  necessary 
steps  in  the  firing  process  or  because  we  cannot  accurately  assess  the  execu- 
tion of  some  of  them  we  ultimately  qualify  a rifleman  on  the  basis  of  whether 
he  hits  the  target. 

In  reflecting  on  the  nature  of  these  three  types  of  tasks  an  important 
implication  emerges  regarding  the  role  of  product  measurement  in  testing 
task  performance:  Because  of  the  interchangeability  of  process  and  products 

for  tasks  of  the  first  two  types,  it  doesn't  really  matter  which  measure  is 
used  to  assess  proficiency;  but  for  tasks  of  Type  3,  product  measurement  is 
very  important.  In  spite  of  this,  in  practice^ performance  tests  for  many  of 
the  latter  type  of  tasks  do  not  attempt  to  measure  product.  Khy  is  this  so? 

Problems  in  Product  Measurement 

The  'easons  largely  stem  from  practical  considerations  in  whi'Ch  the 
measurement  of  task  product  is  viewed  as  either  too  costly,  too  dangerous, 
or  for  other  reasons  simply  too  impractical.  In  testing  such  performances 
as  hand-to-hand  combat,  for  example,  where  task  product  would  take  the  form 
of  disabling  a hostile  enemy,  the  test  developer  is  normally  limited  to 
requiring  the  demonstration  of  task  process.  Similarly,  in  a first  aid  task 
like  controlling  the  bleeding  from  an  external  wound,  the  person  being  tested 
is,  for  obvious  reasons,  asked  only  to  demonstrate  the  process.  Or,  in 
removing  a jammed  round  from  a weapon,  it  is  considered  impractical  to 
actually  jam  a round  in  order  to  create  a valid  test  situation,  so  again 
only  the  steps  in  task  performance  are  measured.  Many  such  examples  are  to 
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be  found  in  the  area  of  interpersonal  behavior.  One  has  to  do  with  instruc- 
tor training,  where  at  least  traditionally  the  military  instructor  trainee 
is  evaluated  by  having  him  prepare  and  deliver  a block  of  instruction  during 
which  he  is  judged  on  such  process  factors  as:  "stood  erect,"  "had  good 

eye  contact  with  audience,"  "could  be  heard  in  the  back  of  the  room,"  "used 
visual  aids  effectively,"  "covered  all  points  in  the  lesson  plan,"  etc. 
Although  the  product  of  instruction  clearly  is  student  learning,  I believe 
it  is  seldom  if  ever  used  as  the  criterion  for  qualifying  an  instructor 
trainee  — probably  because  it  would  involve  a more  time  consuming  and  imprac- 
tical method  of  evaluation.  Another  very  similar  example  which  comes  to 
nind  pertains  to  a recruiter's  task  of  delivering  a persuasive  speech  to  a 
student  audience.  If  the  product  of  this  task  could  be  measured  it  would 
be  in  terms  of  the  number  in  the  audience  who  later  contact  the  recruiter 
with  an  interest  in  enlisting.  But,  again,  because  of  its  implausability 
as  a measure  of  student  achievement,  product  gives  way  to  process  and  the 
recruiter  trainee's  persuasive  speech  is  evaluated  in  much  the  same  way  as 
was  described  above  for  instructor  trainee. 

Dealin^.:,  with  Problems  of  Product  Measurement 

I'm  sure  that  those  of  you  involved  in  perforrance  testing  can  think  of 
many  more  instances  in  which  product  measurement  is  not  used.  Clearly,  some 
of  these  are  justified  by  cost  or  safety  considerations  — but  others  are 
not.  Which  brings  us  to  the  central  point  of  my  remarks  today.  I believe 
that  test  developers  often  fail  to  see  the  importance  of  measuring  task 
outcome;  or  perhaps  they  merely  slight  the  importance  when  faced  with  prac- 
tical limitations  in  its  measurement.  Wliatcver  the  motivation,  I believe 
t..ev  do  not  strive  nearly  hard  enough  to  overcome  resource  problems  which 
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constrain  attempts  to  measure  task  product,  and  too  easily  give  in  to  the 
simplistic  approach  of  measuring  task  process.  The  overriding  question  that 
a test  designer  should  ask  himself  in  this  situation  is:  If  I use  only  a 

process  measure  to  test  a man's  achievement  on  a task,  how  certain  can  I be 
from  this  process  score  that  he  would  also  be  able  to  affect  the  product  or 
outcome  of  the  task?  \Jhere  the  degree  of  certainty  is  substantially  less 
than  that  to  be  expected  from  normal  measurement  error,  the  test  designer 
should  pause  and  reconsider  ways  in  which  time  and  resource  limitations  can 
be  compromised  in  achieveing  at  least  an  approximation  to  product  measurement. 
Although  there  will  remain  instances  in  which  product  measurement  simply 
cannot  be  achieved,  we  will  discover  many  others  where,  through  some  imagina- 
tive thinking,  we  can  devise  simulations  that  will  enable  us  to  assess  task 
outcome  in  a more  relevant  fashion. 

In  testing  the  student  instructor,  for  instance,  I see  no  compelling 
reason  why  we  shouldn’t  get  away  from  the  "charm  school"  approach  to  evalua- 
tion. Why  not  simply  have  him  conduct  a brief  instructional  session  for  a 
small  group  of  students  (perhaps  his  peers) , with  his  achievement  being 
measured  in  terms  of  whether  his  students  have  accomplished  the  instructional 
objective?  In  the  case  of  the  recruiter  trainee's  speech,  evaluating  task 
product  is  more  difficult;  but  surely  a measuie  closer  to  task  outcome 
could  be  achieved  — perhaps  a paid  student  panel  representing  the  potential 
audience  could  be  employed  to  view  and  rate  the  appeal  of  videotaped  trainee 
speeches.  In  zeroing  in  on  critical  motor  skills,  such  as  those  involved 
in  extracting  a jammed  round  from  a weapon  or  in  controlling  bleeding  from 
a wound,  it  would  seem  that  relatively  low  cost  simulators  could  be  devised 
for  use  in  testing  task  outcome.  Hand-to-hand  combat  very  likely  represents 
a case  in  which  ultimate  task  product  simply  cannot  be  measured.  However, 
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in  a similar  vein,  the  Army  is  now  experimenting  with  an  intriguing  method 
of  assessing  the  outcome  of  an  infantry  squad  combat  exercise.  The  principal 
feature  of  the  method  entails  each  participant  having  a number  printed  on 
his  helmet  and  an  Inexpensive  scope  mounted  on  his  rifle;  then,  during  the 
course  of  the  exercise  a soldier  may  "kill"  by  correctly  reporting  an 
enemy's  number,  or  "be  killed"  by  allowing  his  number  to  be  sighted  by  the 
enemy.  Number  size  and  scope  power  have  been  carefully  calibrated  from 
empirical  data  so  chat  the  probability  of  a simulated  "kill"  is  highly 
correlated  with  the  expected  outcome  in  actual  battle.  This  is  an  excellent 
example  of  an  innovative  method  of  achieving  product  measurement  on  a i ask 
that  heretofore  had  been  subject  to  process  evaluation. 

Obviously,  from  these  examples  we  can  see  that  the  accomplishment  of 
product  measurement  is  not  always  a simple  matter;  but  it  is  a demanding 
and  essential  goal  to  be  pursued  by  the  performance  test  developer  if  his 
products  are  to  be  relevant  to  real  world  behavior. 
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BRIEF  OF 

FOUR  RESEARCH  STUDIES  USING 
THE  SELF-EVALUATION  TECHNIQUE 
(SET  STUDIES) 


US  Army  Ordnance  Center  and  School 
Doctrine  and  Training  Development  Division 


Purpose  of  the  SET  Studies 

The  purpose  of  the  four  SET  studies  was  to  determine  if  student  self- 
evaluations  of  their  performance  tests  improved  student  performances  on 
reouired  performance  tests. 

Procedure  of  the  SET  Studies 


This  section  describes  the  subjects,  course  requirements,  testing 
instruments  used,  how  the  tests  were  administered,  and  how  the  scores  were 
obtained. 


The  Subjects 

The  subjects  used  in  these  SET  studies  numbered  three  hundred  and 
fifty-four  students  who  performed  twenty-seven  hundred  and  four  performance 
tests.  These  students  were  enrolled  in  the  44C20  Welder  Course,  41C10/20 
Fire  Control  Instrument  Repairman  Course,  4AE20  Machinist  Course,  and  63G20 
Fuel  and  Elec^tric  Systems  Repairman  Course.  These  courses  were  conducted 
at  the  US  Army  Ordnance  Center  and  School. 

Course  Requirements 

One  of  the  prereouisites  for  each  of  the  courses  is  as  follows: 

Welder,  General  Maintenance  (GM)  score  of  90,  Fire  Control  Instrument 
Repairman,  General  Maintenance  (GM)  score  of  100,  Machinist,  General 
Maintenance  (GM)  score  of  100,  and  Fuel  and  Electric  Systems  Repairman, 
Motor  Maintenance  (MM)  score  of  95.  These  Army  Standard  Scores  are 
derived  from  the  General  Maintenance  and  Motor  Maintenance  Aptitude  Areas 
of  the  Army  Classification  Battery. 

The  Instruments  Used  in  the  Studies 


For  the  evaluation  of  the  apprentice  welders  one  evaluation  form  was 
designed  and  served  the  purpose  of  evaluating  the  eight  performance  tests. 
Any  one  of  the  eight  welds  performed  can  be  written  in  the  blank  space  on 
the  instrument.  A copy  of  this  Instrument  is  attached  as  Appendix  A. 


For  the 
necessary  to 
the  required 
copv  of  this 


evaluation  of  the  Fuel  and  Electric  Systems  Repairman  it  was 
break  down  several  of  the  performance  tests  so  that  some  of 
tasks  could  be  tested  at  a prescribed  testing  station.  A 
instrument  is  attached  as  Appendix  B. 
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For  the  evaUtation  of  the  Fire  Control  Instrument  Repairman  and  the 
Apprentice  Machinist  a separate  performance  evaluation  Instrument  was 
designed  to  cover  each  of  the  performance  tests.  Copies  of  these  Instru- 
ments are  attached  as  Appendix  C and  D. 

Testing  Procedure 

Several  classes  were  used  to  validate  all  self-evaluation  instru- 
ments. After  the  instruments  were  validated,  students  who  did  not  make 
self-evaluations  were  designated  as  the  control  group  and  students  making 
self-evaluations  were  considered  to  be  the  experimental  group. 

The  SET  test  project  director  matched  each  student  in  the  control 
group  with  a comparable  student  in  the  experimental  group  using  the  GM 
or  MM  scores  as  a basis  for  equating  these  matched  pairs.  In  some 
instances  it  was  impossible  to  make  comparable  matches  for  each  of  the 
matched  pairs  when  matching  the  students  by  classes.  The  overall  results 
on  the  total  of  matches  for  each  course  showed  a very  close  mean  for  the 
matched  pairs. 

The  experimental  groups  made  a self-evaluation  for  each  of  the  per- 
formance tests  they  performed.  A technically  qualified  grader  evaluated 
each  performance  test  performed  by  both  the  experimental  and  control 
groups.  These  evaluations  were  forwarded  to  the  SET  study  project  director 
who  scored  the  scl f-evaluator ' s score  sheets,  the  grader's  evaluation  of 
the  se 1 f -eval uator s ' score  sheets  and  the  graders'  evaluations  of  the 
non-self-evaluations.  The  student's  self-evaluation  and  the  grader's 
evaluation  sheets  were  attached  together  and  returned  to  the  students 
with  either  critical  or  complimentary  remarks  which  were  influenced  by 
the  scores  they  made  on  that  performance  test.  No  score  was  recorded 
on  the  returned  student  or  grader's  evaluation  sheets  and  the  comments 
reflected  only  his  weak  or  strong  tasks  on  the  performance  test.  The 
graders'  scores  for  the  non-self-evaluators  were  recorded  by  the  project 
director  opposite  his  match  from  the  self-evaluators. 

The  performance  test  scores  for  each  of  the  matched  pairs  were  used 
to  test  for  a significant  difference  between  the  means  of  the  control 
and  experimental  groups. 

For  the  purpose  of  evaluation  the  SET  studies  were  designed  to  test 
for  significant  difference  between  the  graders'  scores  for  students  who 
made  self-evaluations  and  the  graders'  scores  for  students  who  did  not 
make  self-evaluations. 
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Vhs  secret  of  the  set  studies  which  I am  going  to  present  can  be 
attributed  to  direct  communication  and  automatic  feedback. 


To  show  you  the  Importance  o'”  proper  and  correct  communicative 
procedure  T would  like  to  read  a copy  of  the  registration  form  I filled 
out  when  I registered  for  this  military  testing  association  symposium. 


Now  T will  briefly  cover  the  need  for  conducting  set  studies.  John 
R.  Carrol's  article  ''Neglected  Areas  in  Educational  Research''  presented 
in  number  A2,  May  1961  issue  of  the  Phi  Delta  Kappan  stated  that 
'Research  has  told  us  little  about  the  role  of  motivation  in  school 
learning.''  "Let  us  take  for  granted  that  motivation  is  the  sense  of 
willingness  on  the  part  of  the  learner  to  engage  in  learning.''  Studies 
of  both  intrinsic  and  extrinsic  motivation  are  needed. 

S,  L.  Pressey  and  F.  P.  Robinson  stated  in  their  article  ''Psycholog 
the  New  Education.''  Informing  students  regarding  their  work  is  essentia 
for  intelligent  learning.  If  real  life  tests  are  brought  to  the  point 
where  both  student  and  teacher  can  be  informed  about  progress  in  the 
development  of  social  adjustments,  interests  and  attitudes  this  would 
be  splended. 

The  only  study  which  I could  find  that  even  vaguely  resembled  the 
SET  studies  was  located  in  the  twenty-ninth  yearbook  of  the  National 
Society  for  the  study  of  education  and  was  published  in  1930. 


THE  DESIGN  AND  RESULTS  OF  THE  STUDY  ARE  AS  FOLLOWS: 

Two  equivalent  groups  of  358  Fourth  grade  pupils  were  given  identical 
arithmetic  drill  exercises  for  15  minutes  a week  for  twenty-one  weeks. 

The  only  variable  was  specific  knowledge  of  improvement.  The  members 
of  the  control  group  never  received  their  weekly  scores.  Those  in  the 
experimental  classes  kept  individual  progress  charts  and  pooled  their 
losses  or  gains  in  a graphic  representation  of  improvement  of  the  class 
as  a whole.  The  record  of  actual  improvement  was  made  possible  by  using 
Prill  units  for  whlcl.  comparable  standards  had  been  provided.  This 
scoring  device  compensated  for  difference  in  difficulty  of  the  tasks. 

The  group  which  had  continuous  information  concerning  improvement 
made  significantly  greater  pains  than  the  control  group. 

This  indicates  that  if  students  are  knowledgable  of  what  requirements 
are  expected  of  them  and  they  are  made  aware  of  their  progress  there 
should  be  an  improvement  in  their  overall  performance. 

Dlease  refer  to  the  self  evaluation  Instrument  for  the  welders. 

(Appendix  A) 

Several  classes  were  used  to  validate  this  instrument.  The  first, 
second  and  third  instruments  were  discarded  and  this  instrument  was 


and 
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I adoptpc'  for  the  evaluation.  iiiis  same  instrument  can  be  used  for  all 

I eipht  weldinp  proi«'Cts.  All  that  is  needed  is  to  unite  the  type  of 

I weld  being  performed  in  the  lined  space  provided  on  the  instrument. 

Are  there  any  cue st  ions  .' 

The  same  identical  procedure  was  followed  for  the  next  2 SCI  studies. 

Now  refer  to  the  second  instrument  .'ppendix  h which  is  a sample  of  one 
' of  the  performance  tests  given  to  tie  Fuel  and  Flectric  .'>ystems 

liepairman  course  students,  '’he  development  of  inst  rurient  s to  evaluate  the 
i ’3  matched  pairs  for  7 performance  tests  was  complicated  by  the  number  of  stations 

needed  to  accomplish  the  evaluations.  For  the  seven  tests  administered  a 
total  of  27  sub  tests  were  needed  to  cover  the  22  stat  ions  covered  by  the 
; ~ performance  tests.  It  was  also  difficult  to  set  up  required  standards 

for  each  of  the  tasks  recuired  to  be  performed.  Notice  that  this  was 
solved  bv  the  reference  to  instructor  help.  The  involvement  of  the  instructor 
1 determines  how  well  the  student  could  perform  each  required  task. 

.'.re  tl'ere  anv  curst  ions? 

Please  refer  to  E hr  instrument  for  'he  evaluation  of  the  ^iC  instrument 
repairman.  Appendix  f.  Elpht  periormaii'c  tests  were  developed  with  a 
separate  test  instrument  developed  for  each  performance  test.  The  sample  which 
has  been  furnished  yon  i'  an  example  of  tiie  task  requirements  for  performance 
test  number.  In  i x of  eight  tests  were  easier  to  construct  because  the  .students 
had  to  peri'orm  tasks  to  meet  recuired  standards. 

Are  there  any  questions? 

The  last  samrlc  evaluation  instrument  is  for  the  machinist  course 
and  was  written  to  cover  the  first  performance  test,  Appendix  h.  The  remaining 
F performance  tests  were  written  following  the  same  format,  and  no  problem 
was  '''.countered  since  each  of  the  tests  l-,ad  tolerance  ranges  which  were 
indicative  of  the  type  of  performance  the  student  made. 

I'o  1 have  any  ouest  ions'? 

'"■radinp  of  Instruments 

The  grading  of  the  instruments  is  ver^  simple.  You  simply  take  the 
number  of  tasks  and  multiply  by  For  example  on  the  last  instrument 
vou  '-.ive  10  tasks  x value  ot  U eoual  score  of  AO.  .Suppose  the  student 
'rad  F tasks  in  column  A tasks  in  column  3-9;  and  1 task  in 

column  2 = 2 for  a total  of  '*5;  dividing  A into  1’.3  would  give  a grade 
of  P7.5. 
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riease  refer  to  Appendix  E which  shows  the  final  results  for  the  four 
studies.  It  shows  that  the  self  evaluation  technique  was  successful  for  all 
the  studies  conducted.  A copy  of  the  complete  study  can  be  obtained  upon 
reoues*-  by  writing: 

Dr.  John  J.  Holden 

US  Army  Ordnance  Center  & School 

ATTN:  ATSL-CTD-DT.P 

Aberdeen  Proving  Ground,  Maryland  21005 
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ArPFNPIX  A 


FVAIUATION  CF  STUDENT'S 
WFLDING  PROJECT 


STUDENT'S  NAME 

CLASS  NO  

DATE 


MARK  AN  (X)  IN  THE 
PROPER  SPACE 


STUDENT  SELF-EVALUATION 


GRADER  EVALUATION 


INTRODUCTION:  This  is  a romplete  welding  project  which  will  indicate  how  well 

you  feel  you  performed  the  important  tasks  for  a 

^we  1 d . 

The  purpose  of  this  experiment  is  to  obtain  information  that  will 
help  us  to  improve  the  welding  course.  You  will  not  receive  a 
grade  on  your  welding  project  but  an  honest  evaluation  of  your 
performance  will  be  of  a great  help  to  us. 

DIRECTIONS:  Read  each  Item  on  the  task  list  and  mark  an  (X)  in  the  column 

which  shows  how  well  you  feel  you  performed  that  task. 


Column  Column  3 Column  2 Co  1 umn  1 


TASKS  PFRFCRMFr  • 

Epual  to  Per- 
formance of 
Better  than 
Av'erape  Stu- 
dent ''elder 

Eoual  to 
Performance 
of  Average 
Student 
L’el  der 

Not  quite 
Equal  to 
Performance 
of  Average 
Student 
Welder 

Far  Below 
the  Perfor- 
mance of 
Average 
Student 
We  1 der 

1.  Obtained  correct  flame 
adjustment  (current  setting). 

2.  Obtained  good  fusion 
(tinning  action). 

. Obtained  pood  crown  on 
weld  face. 

4.  Obtained  good  penetration 
(bonding ) . 

i 

5.  Kept  bead  free  of  cracks. 

f.  Obtained  even  uniform 
head  shape. 



7.  Kept  bead  free  of  under- 
cut (droplets). 

P.  Kept  head  free  of  over- 
1 ap . 

Kept  bead  free  of  burned 
or  crystalllred  metal. 

10.  Kept  bead  free  of  holes. 

< 

Could  finish  proiert  In  time  allotted  YES  KC 
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APPENDIX  B 


PERFORMANCE  TEST  63C20 
DELIVERY  VALVE  SPRING  TEST 
STATION  1 Checkinp  Delivery  Valve 

Student's  Name EJ  Student  Self-Evaluation 

na..  Date £7  Instructor  Evaluation 


1.  Connected  the  nozrle  tester  and  prepared  the  pump  for  testing  the  delivery 
valve  springs  with: 

CJ  No  trouble  o Little  trouble  a Lots  of  trouble 

Received  instructor's  help  to  perform  operation:  / / 'DS  / / NO 

If  VrS  is  checked,  how  manv  times  did  the  instructor  help?  j / 

?,  Observed  opening  pressure  of  all  delivery  valve  springs,  recorded  readings 
and  determined  condition  of  valves: 

j—j  f valves  correctly  4 or  5 valves  correctly  3 or  less  valves 

correctly 


Received  Instructor's  help  to  perform  operation:  n 

If  YES  is  checked,  how  manv  tines  did  the  instructor  help? 


NO 


. Overall  performance  for  this  station: 

o Above  average  student  performance  / ) Ei^ual  to  average  student 
performance  o Felov  average  student  performance  o Far  below  average 
student  performance 


APPENDIX  P (Cent) 


PERFORMANCE  TEST  f3G‘iO 

STATION  2 Timlnp  APF6RR  Injector  Pump  to  the  Mack  ENDT-673  Engine 

Student's  Name CJ  Student  Self-Evaluation 

Class  Date n Instructor  Evaluation 


1.  Positioned  number  1 cylinder  in  correct  position  on  compression  stroke  with 
timing  mark  aligned  to: 

£7  ?9°  RTDC  £7  38°  or  30°  RTDC  £7  27°  or  31°  BTDC 

Received'  instructor's  help  to  perform  operation:  ! 7 YES  [~1  NC 

If  YES  is  checked,  how  many  times  did  the  instructor  help?  / / 

Secured  pump  drive  coupler  to  engine  with  pump  camshaft  at  ''part  closing" 
position  for  number  1 outlet  with: 

o No  trouble  £7  Little  trouble  ! 7 Lots  of  trouble 

Received  instructor's  help  to  perform  ooeration:  / 1 YES  o NO 

If  YES  is  checked,  how  many  times  did  the  instructor  help?  / / 

■* . Replaced  all  lines,  covers,  and  valves  and  engine  started  with: 

£7  No  trouble  / 7 Little  trouble  £7  Didn't  start 

Received  instructor's  help  to  perform  operation:  / YES  o NO 

If  YES  is  checked,  how  many  times  did  the  instructor  help?  F / 

Overall  performance  for  this  station: 

t 7 Above  average  student  performance  /~~7  Equal  to  average  student 
performance  [~~1  Relow  average  student  performance  £7  Ear  below 
average  student  performance 
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APrrNTIX  C (Cont) 

PERFORMANfF  TFST  63G20 
STATION  3 Tlminp  the  Unit  Injectors 

Student's  Name /~~7  Student  Self-Evaluation 

D Instructor  Evaluation 

1.  Positioned  enfine  for  tlminp  of  the  selected  injector  correctly: 

o No  trouble  / 7 Little  trouble  CZJ  Lots  of  trouble 

Received  instructor's  help  to  perform  operation:  CU  YES  [ZJ  NO 

If  YES  is  checked,  how  many  times  did  the  instructor  help?  / / 

2.  Selected  correct  timing  gape  and  adjusted  injector  to  specifications  with: 

o No  trouble  o Little  trouble  o Lots  of  trouble 

Received  Instructor's  help  to  perform  operation:  o YES  o NO 

Tf  YES  is  checked,  how  many  times  did  the  instructor  help?  / / 

. Overall  performance-  for  this  station: 

D Above  average  student  performance  /~~7  Equal  to  average  student 

performance  o Below  average  student  performance  ! 7 Far  below 

average  student  performance 

! Rid  you  have  enough  time  to  complete  rhe  test  required  at  each  of  the  3 stations.’ 

' n YES  /~7  NC . If  answer  is  NO,  write  reason(s)  on  bac  k of  this  sheet . 

i 

i 


» >1  </  M 


■ ^ it 
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APPENDIX  C 


SIXTH  PERFORMANCE  EXAMINATION  41CI0/20 
Maintenance  and  Adjustments  of  Panoramic  Telescope  Ml  15 

MARK  AN  (X)  IN  THE 

STUDENT'S  NAME PROPER  BOX 

CLASS  NUMBER STUDENT  SELF-EVALUATION  | | 

DATE GRADER  EVALUATION  | 1 

PURPOSE:  The  purpose  of  this  self-evaluation  experiment  is  to  obtain  information 

that  will  help  us  to  improve  the  fire  control  instrument  repair  course. 
You  will  not  receive  a grade  on  your  self-evaluation  for  this  project 
but  an  honest  evaluation  of  your  performance  will  be  of  a great  help  to 
us  to  evaluate  the  course.  Your  course  grade  will  be  determined  by  the 
score  you  obtain  from  the  department’s  performance  examination. 

DIRECTIONS:  Mark  a / in  the  TOLFRANCE  RANGE  COLUMN  for  the  TOLERANCE  which  you 

obtained  on  each  of  the  listed  dimensions  to  be  checked. 


TOLERANCE  RANGE 


DIMENSIONS 
TO  BE  CHECKED 

Column  4 

Column  3 

Column  2 

Column  1 

Perfect 

Within 

Within 

Over 

1. 
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Fair 
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APPENDIX  D 

FIRST  PERFORMANCE  EXAMINATION 
EVALUATION  OF  STUDENT'S 
MACHINIST  PROJECT 
SHAPER  PERFORMANCE  TEST 

MARK  AN  (X)  IN  THE 

STUDENT  ' S NAME PROPER  BOX 

CLASS  NUMBER STUDENT  SELF-EVALUATION 

PATF GRADER  EVALUATION  nn 

PURPOSE:  The  purpose  of  this  self-evaluation  experiment  is  to  obtain  information 

that  will  help  us  to  improve  the  machinist  course.  You  will  not  receive 
a grade  on  your  self-evaluation  for  this  project  but  an  honest  evalua- 
tion of  your  performance  will  be  of  a great  help  to  us  to  evaluate  the 
course. 

DIRECTIONS:  Make  a ^ in  the  TOLERANCE  RANGE  COLUMN  for  the  TOLERANCE  which  you 

obtained  on  each  of  the  listed  dimensions  to  be  checked. 


DIMENSIONS 
TC  BE  CHECKED 

DIMENSION 
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□ 

CD 
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CD 
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□ 

□ 

CD 
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□ 

□ 

□ 

O 
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Step 
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CZ] 

□ 

□ 

□ 

5.  Width  of  Second 
Step 

6.  Length  of  Block 
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a 
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a 

□ 

4 1/32” 
□ 

□ 

4 3/64” 

CD 

CD 

Above  or 
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CD 

7.  Linear  Pitch 
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o 

nj 

CD 

CD 

d.  Width  of  V 

1 - 3/16" 

□ 

HD 

CD 

CD 

9.  Depth  of  Teeth 

0.270” 

0.005 
- 0.000 

0.006 
- 0.001 

■t  0.007 
- 0.002 

Above  4 0.007 
Below  - 0.0C2 

□ 

□ 

□ 

□ 

Better  Than 
Average 

Average 

Below 
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Far  Below 
Average 

10.  General 
Appearance 

CZI 

CD 

CD 

CD 

Could  you  finish  project  in  time  allotted  YDSi  l NG|  |.  If  answer  is  NO  tell  why 
you  didn't  have  enough  time.  Did  you  finish  ahead  of  time  YE S|  1. 
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l^hULATTG.;r  OP  r.ET  STUDY  FUNDIN’ 
KCK  177  MATCilEL  PATTS  OF  STUFFN 
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INTRODUCTION 


A new,  more  specific  language  for  describing  work  activities  is  being 
designed  for  the  Army  on  an  experimental  basis. 1 The  new  language  is 
based  upon  a concept  called  the  "duty  module".  Duty  modules  are  clusters 
of  tasks  that  tend  to  go  together  occupationally  and  organizationally  in 
meaningful  ways. 

The  need  to  evaluate  the  feasibility  of  personnel  information  systems 
based  upon  clusters  of  tasks  smaller  than  a Military  Occupational  Specialty 
(MOS)  was  originally  suggested  by  personnel  at  the  Army  Research  Institute. 
The  development  of  the  duty  module  concept  was  a team  effort  involving 
staff  members  from  the  Army  Research  Institute  (ARI)  and  the  American 
Institutes  for  Research  (AIR). 2 

Duty  modules  arc  rationally  derived  clusters  of  work  activity  based  upon 
a detailed  examination  and  grouping  of  task  inventory  or  job  analysis  data. 
Attention  is  then  given  to  ways  in  which  these  tentatively  identified  job 
content  modules  can  be  tested  against  various  available  criteria  of  opera- 
tional utility.  One  relevant  consideration  is  whether  job  content  modules 
can  be  used  as  field  assignment  modules.  Another  possible  application  is  in 
the  area  of  requirement  planning  and  unit  effectiveness.  A source  of 
information  here  is  data  that  can  be  gathered  in  conjunction  with  unit 
training  and  unit  effectiveness  exercises  that  are  performed  in  the  field. 

The  word  "module"  was  chosen  because  job  activity  clusters,  like 
equipment  components  of  the  same  name,  are  meant  to  be  largely  self- 
contained,  independent  units  of  work.  For  purposes  of  occupational  class- 
ification, a duty  module  is  a cluster  of  tasks  that  apply  without  modifica- 
tion in  a number  of  occupational  classifications  or  specialties. 


The  work  was  carried  out  under  Contract  No.  DAHC- 19-71 -C-0004 , "A 
Taxonomic  Base  for  Future  Information  and  Decision  Systems",  and  Contract 
No.  DAHC- 19-73-C-0041 , "A  Comparison  of  Officer  .lob  Content  Modules  with 
Activity  Grouping  Implicit  in  Course  I'esign",  awarded  by  the  Army 
Research  Institute  to  the  American  Institutes  for  Research. 


Key  personnel  at  ARI  wore  Mr.  Cecil  Johnson,  who  provided  the  initial 
guidance  as  the  Contract  Officer's  Technical  Representative,  Dr.  J.E. 
Uhlaner,  Technical  Director,  and  Mrs.  Bertha  Cory,  who  succeeded  Mr. 
Johnson  as  the  Technical  Representative. 

Key  AIR  staff  members,  in  addition  to  Dr.  Robert  W.  Stephenson,  who 
was  Project  Director,  included  Dr.  Robert  Miller,  who  served  as 
Consulting  Scientist,  Colonel  Warren  P.  Davis,  Mr.  Harry  1.  Hadley, 
Dr.  Edwin  A.  Fleishman,  Dr.  Albert  S.  Glickman,  Mr.  Clifford  P.  Hahn, 
Dr.  Ronald  P.  Carver,  and  Mr.  Albert  Farina. 


270 


Another  form  of  construct  validation  is  available  through  examination 
of  mission  elements  which  are  activities  that  are  used  to  design  and  evaluate 
the  performance  of  organizational  unit. 

This  presentation  will  describe  two  different  kinds  of  modular  evalua- 
tion devices  --  sets  of  tasks  performed  by  individuals,  and  sets  of  tasks 
performed  by  organizational  units. ^ 

INDIVIDUAL  PROFICIENCY  TESTS 

Every  individual  proficiency  test  and  every  Army  training  test  is 
already  divided  into  special  component  sections  with  separate  scores.  Be- 
fore going  into  detail  about  what  modular  component  scores  are  supposed  to 
do  and  supposed  to  look  like,  it  is  necessary  to  describe  what  these  exist- 
ing systems  are  like. 

ENLISTED  PERSONNEL  PERFORMANCE  EVALUATION 

This  discussion  should  be  prefaced  by  noting  that  evaluation  of  the 
performance  of  enlisted  personnel  is  an  important  responsibility  of  every 
commissioned  and  senior  noncommissioned  officer  in  the  Army.  The  rewards 
and  punishments  associated  with  such  evaluations  give  commissioned  and 
noncommissioned  officers  the  necessary  control  over  enlisted  personnel  to 
maintain  and  improve  effectiveness.  In  addition  to  this  important  super- 
visory function,  however,  there  are  a number  of  formal  proficiency- evalua- 
tion procedures  for  enlisted  personnel  that  are  conducted  by  various 
headquarters.  The  most  important  of  these,  for  purposes  of  this  paper, 
is  the  U.S.  Army  Enlisted  Evaluation  Center,  located  at  Fort  Benjamin 
Harrison,  Indianapolis,  Indiana. 


These  developments  are  described  in  detail  in  Miller,  R.B.  A Taxonomic 
Base  for  Future  Management  Information  and  Decision  Systems:  Theoretical 

Background  to  the  Design  of  Duty  Modules;  American  Institutes  for 
Research,  Washington,  D.C.,  Technical  Report  A1R-23500-7/71-TR-2 , July 
1971.  (U.S.  Army  Behavior  and  Systems  Research  Laboratory,  BESRL  Technical 

Research  Note,  in  preparation.);  and  Stephenson,  R.W.  (American  Institutes 
for  Research,  Washington,  D.C.)  A Taxonomic  Base  for  Future  Management 
Information  and  Decision  Systems:  A Common  Language  for  Resource  and 

Requirement  Planning;  U.S.  Army  Behavior  and  Systems  Research  Laboratory, 
Arlington,  Va. , Technical  Research  Note  244  (AD-757-794) , October,  1972. 


The  Enlisted  Evaluation  Center  is  the  major  operating  element  of  the 
formal  enlisted  evaluation  system.  It  was  established  in  1958  as  a Class 
2 activity  of  the  Army.  Its  primary  purpose  then  was  to  help  the  Army 
manage  the  proficiency  pay  program,  which  had  been  established  in  response 
to  the  recommendations  of  the  Cordiner  Committee — the  Defense  Advisory 
Committee  on  Professional  and  Technical  Compensation--in  1956  and  1957. 

Monetary  incentives  were  one  of  the  Committee's  proposals  designed 
to  improve  personnel  retention  and  job  motivation  among  trained  technical 
specialties  and,  at  the  same  time,  stimulate  higher  quality  performance 
among  all  enlisted  personnel.  Proficiency  pay,  as  a concept,  emanated 
from  this  Committee  recommendation.  However,  an  underlying  principle  of 
this  concept  was  that  proficiency  pay  must  be  directly  related  to  the 
demonstrated  level  of  proficiency  and  must  be  contingent  upon  periodic 
checks  to  ensure  maintenance  of  that  proficiency.  The  Army  enlisted 
evaluation  system  was  developed  to  meet  this  requirement. 

ARMY  ENLISTED  EVALUATION  SYSTEM 

The  enlisted  evaluation  system  consists  of  two  major  components: 

(a)  evaluation  of  the  enlisted  man's  knowledge  of  the  various  duties  that 
are  required  at  his  skill  level  in  his  MOS,  as  indicated  by  MOS  evaluation 
tests  and  performance  tests;  and  (b)  evaluation  of  performance  in  the 
currently  assigned  duty  position,  as  indicated  by  supervisory  ratings  on 
the  enlisted  evaluation  report  (see  Figure  1) . A rating  system  is  applied 
to  the  scores  obtained  on  these  instruments,  and  it  is  used  to  compute  a 
composite  score  for  taking  individual  personnel  actions.  This  MOS  evalu- 
ation score  indicates  the  individual's  relative  standing  among  thcfSe  evalu- 
ated in  the  same  MOS  and  skill  level  and  in  the  same  pay  grade.  It  is 
used  to  verify  MOS  qualification,  to  assist  in  determining  promotion  eli- 
gibility, to  award  proficiency  pay,  to  guide  remedial  training,  and  in  a 
variety  of  other  personnel  actions. 


PROFICIENCY  TESTING  "AREA  SCORES" 


The  characteristics  of  the  MOS  proficiency  testing  program  will  not 
be  detailed  herein,  but  one  particular  aspect  of  the  MOS  proficiency  test 
program  directly  relevant  to  this  paper  will  be  considered--the  MOS  "major 
area"  scores.  Each  MOS  evaluation  test  is  organized  into  six  to  nine 
major  areas;  that  is,  six  to  nine  subscores.  The  six  rtiajor  areas  for  an 
Infantry  senior  sergeant,  for  example,  are  weapons,  tactics,  field  activi- 
ties, unit  defense,  administration,  and  personnel  accounting.  Study 
references  from  Army  regulations,  pamphlets,  field  and  technical  manuals, 
and  other  manuals  are  coded  to  each  of  the  major  areas  in  an  accompanying 
study  guide  so  that  eacli  soldier  can  locate  the  printed  materials  upon 
which  tlie  test  is  based.  He  may  study  these  reference  materials  to  im- 
prove his  knowledge  and  performance. 
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TEST  EEU  RAWCOM5*OSlTE  EVALUATION 

SCORE  SCORE  SCORE  SCORE 


The  major  areas  are  weighted  according  to  the  relative  importance 
of  the  functions  in  the  missions  of  all  units  that  are  authorized  duty 
positions  in  the  MOS  skill  level,  and  not  on  the  basis  of  the  time  re- 
quired to  teach  the  subject  matter  in  formal  classroom  courses  nor  on 
the  basis  of  the  number  of  personnel  assigned  or  authorized  for  specific 
duty  positions.  The  number  of  questions  allocated  to  an  area  out  of  the 
total  number  of  items  in  an  MOS  proficiency  test  indicates  the  weight 
assigned  to  that  area.  Subscores  for  these  major  areas  are  useful  not 
only  to  the  soldiers  tested,  who  can  use  the  information  to  improve  their 
performance,  but  also  to  various  headquarters,  to  centralized  management 
programs,  and  to  commanders  for  managing  assignments  and  training  pro- 
grams . 


DIFFERENT  APPROACHES  TO  DUTY  AREAS 


Unfortunatel)',  there  is  no  consistent  theoretical  basis  or  consistent 
approach  to  the  definition  of  these  area  scores  by  test  developers,  train- 
ing personnel,  or  requirement  planners  (see  Table  1).  Some  of  the  areas 
for  which  the  Unlisted  evaluation  Center  has  developed  scores  can  be 
classified  as  duty  areas.  These  area  scores  roughly  correspond  to  subject 
matter  areas  within  an  MOS.  They  are  used  to  provide  information  as  re- 
gards an  enlisted  man's  strengths  and  weaknesses  in  selected  subject 
areas,  and  they  are  associated  with  specific  subject  matter  references. 


Table  1.  Different  Approaches  to  Duty  Areas 


Approach 

MOS  proficiency  tests 
Army  training  schools 

Requirement  planners 


Duty  Areas 

Selected  by  test  developers 

Identified  by  systems 
engineering  of  training 

Associated  with  additional 
skill  identifiers 


If  subject  matter  references  happen  to  be  organized  in  terms  of  duty 
areas,  it  is  easier  to  find  the  appropriate  references  that  need  to  be 
studied;  however,  it  is  not  essential.  A sample  list  of  major  areas  in 
an  MOS  is  given; 

1 Weapons 

2 Tactics 

5 Field  Activities 

4 Unit  Defense 

5 Administration 

6 Personnel  Accounting 

The  study  guides  list  the  regulations  and  technical  manuals  for  the  vari- 
ous areas,  and  no  great  amount  of  effort  is  needed  to  find  the  references 
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that  correspond  to  the  area  in  which  a low  score  was  received  in  the 
proficiency  evaluation  test  (see  Table  2). 


Table  2,  Sample  Study  Guide 

Maj  or 


References  Area 


Army  Regulations 

65-75  5 

210-10  6 

DA  Pamphlets 

600-8  6 

672-2  3 

Field  Manuals 

5-15  4 

7-10  2 


It  should  be  clear  that  the  rise  of  area  scores  for  proficiency  tests 
was  an  important  development  in  the  design  of  MOS  proficiency  tests.  Such 
feedback  systems  are  an  integral  part  of  any  sophisticated  testing  program. 
What,  then,  would  be  questioned  in  the  design  of  these  area  scores  as  they 
are  used  by  the  proficiency  testing  system?  The  point  raised  is  not  so 
much  how  the  proficiency  testing  subsystem  works,  but  the  manner  in  which 
these  area  scores  interface  with  other  personnel  subsystems  in  the'' Army. 

The  Army  training  schools,  for  example,  identify  major  duty  areas  at 
great  expense  and  with  great  difficulty,  as  part  of  their  systems  en- 
gineering of  training  process  (see  Figure  2).  Systems  engineering  of 
training  is  a long,  drawn-out  procedure,  involving  detailed  job  analysis 
and  the  application  of  systems  engineering  principles  and  approaches  in 
order  to  break  a job  into  components  and  then  select  components  for  train- 
ing. The  job  is  also  organized  into  "areas"  when  a Program  of  Instruction 
(POI)  is  prepared.  There  is  no  consistent  relationship  between  these  POI 
area  scores  designed  by  the  training  people  and  the  area  scores  as  used 
by  the  proficiency  training  people. 

Requirement  planners  are  also  interested  in  a different  kind  of  duty 
area.  For  example,  the  Army  has  additional  skill  identifiers  (ASI)  that 
are  authorized  for  functional  skills,  which  are  not  consistently  required 
of  all  the  job  incumbents  in  an  MOS.  An  example  of  such  an  additional 
skill  identifier  is  the  ability  to  maintain  a specific  type  of  system 
(e.g. , maintenance  on  the  Hawk  Guided  Missile  Simulator,  or  the  ability 
to  work  with  specially  trained  scout  dogs).  These  different  approaches  to 
duty  areas  are  not  necessarily  incompatible  with  each  other,  but  they  are 
all  different.  When  you  have  three  different  parts  of  the  same  organi- 
zation --  and  the  Army  is  one  organization  --  using  three  completely  different 
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Figure  2.  ’Simplified  Flow  Process  of  Systems 
Engineering  of  Training.^ 

^From  Southeastern  Signal  School  Briefing  Supplement. 
Systems  engineering  of  training  at  USASESS.  Undated. 
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language  systems  to  describe  the  same  kind  of  work,  it  is  likely  that 
there  will  be  some  unnecessary  duplication  of  effort.  This  could  be 
avoided  if  a common  language  could  be  designed  for  all  three  parts  of 
the  organization  (e.g.,  the  Army)  to  use. 


THE  DUTY  MODULE 


A duty  module  is  a group  of  occupationally  interrelated  tasks  smaller 
than  an  occupational  specialty.  It  is  modular  in  the  sense  that  it  can 
be  used  as  a plug-in  unit  to  a variety  of  different  occupational  special- 
ties. Table  3 defines  the  module  group,  number,  and  title.  Table  4 
shows  an  MOS  duty  module  matrix  for  Army  military  occupational  special- 
ties. As  one  can  see,  a relatively  small  number  of  duty  modules  can 
account  for  seven  different  MOS.  Notice  that  each  of  these  military 
occupational  specialties  has  demonstrable  similarity  with  other  MOS. 


Table  3.  Module  Group,  Number,  and  Title 

A ADMINISTRATION 

A-1  Performs  general  administration  at  unit  level 

A-2  Performs  unit  supervision  and  control  of  personnel  ^ 

A-3  Establishes  and  operates  a unit  mail  room 
A-4  Types,  files,  and  performs  general  clerical 
operations 

B TRAINING  j 

B-1  Conducts  or  participates  in  unit  and  individual 

training  : 

C COMMUNICATIONS  I 

C-1  Operates  unit  tactical  communications  equipment 
(excluding  use  of  Morse  code) 

C-2  Installs  and  maintains  unit  tactical  wire 
communication  systems 

D TRANSPORTATION  i 

D-1  Operates  unit  combat  support  vehicles 

i 

(continued) \ 


■ - 


Table  3 (continued) 


E TACTICAL  OPERATIONS 

E-1  Prepares  and  employs  maps,  charts,  and  instruments 
in  land  navigation 

E-2  Engages  enemy  with  tank  and  Armor  vehicle 
mounted  assault  weapons 

E-3  Drives  tanks  and  associated  Armor  combat  vehicles 

E-4  Emplaces,  reports,  and  neutralizes  tactical  obstacles 

E-5  Performs  in  mounted,  dismounted,  airborne  or  long- 
range  patrols 

E-6  Engages  enemy  with  mortars 

E-7  Participates  in  ground  tactical  operations  as 
member  of  a maneuver  unit 

E-9  Engages  enemy  in  close  combat  with  individual 
weapons  and  machine  guns 

E-10  Engages  enemy  with  recoilless  rifles  and  direct 
fire  missiles 

E-11  Functions  under  CBR  warfare  conditions 


F STAFF  MANAGEMENT 

F-1  Performs  tactical  operations  support  duties 
F-2  Performs  tactical  intelligence  support  duties 


G MAINTENANCE 


G-1  Performs  user  maintenance  on  individual  and  unit 
equipment  and  weapons  (excluding  motor  vehicles) 
G-2  Performs  organizational  maintenance  on  track 
and  wheel  vehicle  mechanical  systems 
G-5  Performs  organizational  maintenance  on  track 
and  wheel  vehicle  electrical  systems 
G-4  Performs  maintenance  administration 


11  FOOD  SERVICE 

11-1  Establishes  and  operates  a field  mess 
11-2  Prepares  and  serves  meals 

11-5  Operates  a mess  facility 

I SUPPLY 

I-l  Establishes  and  operates  a unit  supply  point 


J PERSONNEL 

J-1  Initiates,  posts,  files,  and  retrieves  information 
from  personnel  records 

J-2  Manages  individual  enlisted  personnel  and  carries 
out  manpower  and  personnel  management  programs 
,1-3  Processes  personal  affairs  actions  for  individuals 
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Table  4.  MOS  — Duty  Module  Matrix 


Duty  ^ 
Modules 

IIB 

Military  Occupational 
lie  IIP  HE 

Specialties 
IIF  IIG 

IIH 

A-1 

X 

A-2 

X 

X 

X 

X 

X 

X 

A- 3 

X 

X 

X 

B-1 

X 

X 

X 

X 

X 

X 

C-I 

X 

X 

X 

X 

X 

X 

U 1 

X 

X 

X 

X 

t-1 

X 

X 

X 

X 

X X 

X 

b-2 

X 

F';-3 

X 

X 

h-4 

X 

X 

X 

X 

H-5 

X 

X 

X 

X 

X 

X 

E-6 

X 

E-7 

X 

X 

X 

X 

X X 

X 

E-9 

X 

X 

E-10 

X 

F-1 

X 

X 

X 

I- -2 

X 

X 

See  Table  3 for  definitions  of  duty  modules. 


It  is  also  possible  to  use  duty  modules  to  express  personnel  require- 
ments. The  list  of  work  activities  in  Table  5 has  not  been  formally  ap- 
proved, but  it  illustrates  the  type  of  approach  that  can  be  used.  Given 
a data  processing  group,  there  is  a need  to  supervise,  to  plan  the  analy- 
sis of  the  reporting,  to  keypunch,  and  so  on.  The  number  of  full-time 
duty  positions  needed  in  the  organization  is  ten.  You  can  also  specify 
the  requirements  in  terms  of  the  number  of  people  qualified  to  perform 
each  work  activity.  If  you  have  a computer  activity  with  a lot  of  night 
shift  work,  you  are  going  to  need  at  least  three  people  who  can  supervise. 
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Table  5.  Work  Activity  Requirements 


Work  Activity 

Minimum  Number  of 
People  Needed 
with  This  Skill 

Number  (or 
Proportion)  of 
Full-Time  Duty 
Positions 

Supervision 

5 

1.00 

Planning  of  analysis 
and  reporting 

3 

.25 

Receipt  and  verification 
of  input  data 

3 

1.00 

COBOL  programming 

2 

2.00 

Keypunching 

8 

1.50 

Computer  operation, 
including  peripherals 

5 

3.00 

Interpretation  of  output 

1.00 

Preparation  of  reports 

2 

.25 

Total  Number  of  Work 
Activity  Requirements 

50 

Minimum  Number  of 
People  Required 

10.00  '■ 

You  may  want  one  or  two  people  for  backup,  in  case  of  illness  or  vacations 
Even  though  there  may  only  be  three  duty  positions  involving  supervision, 
you  may  want  five  people  to  be  qualified  as  shift  supervisors.  Similarly, 
in  planning  the  analysis  and  the  reporting,  you  may  want  three  qualified 
persons,  but  it  is  only  a quarter-time  job.  In  other  words,  defining  work 
activities  in  terms  of  duty  modules  can  provide  a more  efficient  use  of 
personnel  with  all  functions  covered,  using  a minimum  number  of  personnel. 


TESTS  OF  EFFECTIVENESS  OF  DUTY  MODULES 


Resource  and  requirement  planning  experts  must  first  agree  upon  the 
qualification  requirements.  Secondly,  compatibility  with  work  practices 
in  the  field  is  involved.  For  this,  actual  survey  data  regarding  the  way 

in  which  tasks  are  assigned  in  the  field  can  be  employed.  A third  test 
is  to  evaluate  the  usefulness  of  the  module  in  planning  and  evaluating 
the  requirements  for  and  performance  of  organizational  units. 
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TEST  1 


A first  test  is  to  ask  experts  to  design  job  content  modules.  Typi- 
cal job  content  modules  are: 

(1)  Operates  unit  tactical  communications  equipment, 
excluding  Morse  code. 

(2)  Installs  and  maintains  unit  tactical  wire  communi- 
cation system. 

In  this  case,  the  people  who  are  asked  to  operate  unit  tactical  communi- 
cations equipment  arc  usually  different  from  those  who  install  and 
maintain  it.  Neither  of  these  work  activities,  however,  is  a full-time 
position  for  anyone.  These  are  modular  things  that  can  be  assigned  to 
different  people. 


TEST  2 

A second  test  of  duty  modules  is  compatibility  with  assignment 
practices  in  the  field.  Some  data  have  already  been  analyzed  (see  Figure 
5).  Data  were  used  that  were  already  in  existence,  and  that  had  been 
collected  with  task  inventories  that  were  administered  by  the  Army  Office 
of  Personnel  Operations.  The  data  base  is  called  the  Military  Occupations 
Data  Bank,  or  MODE.  The  original  task  statements  in  MODE  are  organized 
in  terms  of  functional  areas  of  performance  (see  the  administration  and 
training  columns  on  the  right  side  of  Figure  3) . The  rows  correspond  to 
task  clusters  that  were  identified  in  an  empirical  clustering  of  tasks. 
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The  Comprehensive  Occupational  Data  Analysis  Programs  (CODAP) 
system,  developed  by  the  Air  Force,  was  examined  to  determine  it  applicability. 
TTiis  system  was  developed  to  cluster  people  rather  than  tasks,  however.  The 
Army  Research  Institute  modified  the  CODAP  system  to  the  problem  at  hand. 

People  were  not  clustered  together  on  the  basis  of  their  similarity  in  terms 
of  task  performance,  instead,  tasks  were  clustered  together  on  the  basis  of 
the  probability  that  the  tasks  would  be  assigned  to  the  same  people.  On  the 
left  side  of  Figure  3,  clustering  is  compared  with  some  duty  modules  that  were 
analyzed  in  terms  of  the  "Obverse  Cluster  Analysis"  system  designed  (see 
Table  3 for  definitions  of  duty  modules).  The  computer  run  suggested  that 
there  were  seven  task  clusters  in  this  particular  group  of  tasks.  The  percen- 
tages shoi'm  in  the  duty  module  columns  on  the  left  side  of  Figure  3 indicate 
the  percent  of  tasks  in  each  of  the  empirically  identified  clusters  that  fall 
into  duty  module  categories  A-1  through  B-2.  The  percentages  shown  on  the 
right-hand  side  of  the  figure  indicate  the  percent  of  tasks  in  the  empirically 
identified  clusters  that  fall  into  each  of  the  administrative  areas  used  to 
group  tasks  in  the  Military  Occupations  Data  Bank. 

It  is  important  to  note  that  the  design  of  the  duty  modules  is  not 
complete,  nor  is  the  preparation  of  the  task  statements.  These  task 
statements  will  be  revised  to  reflect  additional  job  data  and  evidence 
concerning  the  organization  and  application  of  command  authority  in  field 
units.  Task  statements  will  also  be  revised  as  the  duty  module  system  is 
further  developed. 

It  does  not  necessarily  follow  that  tlicre  will  be  complete  agreement 
with  the  computer  runs.  For  example,  it  is  possible  that  the  first  three 
empirical  task  clusters  would  be  considered  as  really  one  cluster  rather  than 
three.  Before  that  conclusion  can  be  made,  many  other  occupational  specialties, 
in  addition  to  the  one  that  these  data  are  based  upon,  will  have  to  be  re- 
viewed. Essentially,  duty  modules  are  derived  from  many  different  occupa- 
tional specialties  rather  than  just  one,  which  is  the  case  with  this  particular 
computer  run.  The  decision  as  to  whether  these  first  three  clusters  are  one 
cluster  or  two  or  three  clusters  must  be  based  upon  data  in  other  occupational 
specialties  as  well. 


TEST  3 

A third  test  of  the  utility  of  duty  modules  is  the  applicability  to  the 
evaluation  of  unit  performance  in  the  field.  Task  checklist  items  have  been 


Bottenberg,  R.  A.,  and  Christal,  R.  E. , "An  iterative  technique  for  cluster- 
ing criteria  wliich  retains  optimum  predictive  efficiency".  The  Journal  of 
Experimental  Education,  36,  (4),  Summer,  1968,  28-34. 

Edison,  W.  , and  Eastman,  R.  T. , "Cluster  analysis  of  occupational  data  with 
focus  on  task  ratlier  than  people",  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  Research  Memorandum,  November,  1973. 
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devised  based  upon  the  duty  modules.  Each  duty  module  consists  of  a collec- 
tion of  task  statements.  A study  of  Army  Training  Tests  (ATTs)  revealed  that 
task  statements  could  be  converted  into  checklist  items.  This  provides  the 
means  by  which  information  can  be  collected  about  the  performance  of  indivi- 
duals during  unit  training  tests.  Separate  scores  are  obtained  for  separate 
practice  maneuvers.  Separate  scores  can  be  obtained  for  movement,  defense, 
and  attack,  as  noted  in  Figure  4.  Each  exercise  will  have  a scenario,  the 
possibility  of  casualties,  and  so  forth.  Duty  modules  will  be  looked  at  during 
the  appropriate  phase  of  the  Army  Training  Tests.  They  would  not  be  tested 
during  every  single  phase. 


Unit  Test  Phase 


Module 

Task  Checklist  Item 

Movement 

Defense 

Attack 

C-1 

Establish  and  operate 
field  communications 
relay  station 

X 

X 

G-1 

Make  entries  in 

equipment  log  books 

X 

X 

Figure  4.  Step  3,  Relationship  to  Unit  Performance. 


SUMMARY 

These  individual  duty  modules  are  designed  to  be  derived  from  a variety 
of  specialties  rather  than  just  one.  They  are  economical  in  the  sense  that 
many  different  task  inventories  can  be  designed  and  jobs  described  with  a 
small  number  of  duty  modules.  They  can  utilize  task  inventory  data  already 
collected  on  assignment  and  work  assignment  practices  in  the  field.  They 
meet  an  apparent  need  for  consistency  at  a level  of  generality  between  the  MOS 
and  the  task,  at  an  optimum  level  of  detail.  Ultimately,  if  duty  modules  can 
provide  a language  acceptable  to  both  resources  planners  and  requirements 
planners,  they  will  facilitate  communication  and  promote  better  matching  of 
men  and  jobs. 


PERFORMANCE  STANDARDS  AND  SKILL  LEVELS 


Tt  is  not  possible  to  talk  about  testing  in  terms  of  duty  modules  with- 
out first  talking  about  testing  standards.  Performance  standards  in  the 
Army  are  prepared  systematically  as  part  of  the  systems  engineering  of  train- 
ing process  for  the  design  of  Army  school  courses.  At  one  point  in  systems 
engineering,  task  and  skill  analysis  sheets  are  prepared.  These  result  in 
evaluation  plans.  It  is  a standard  practice  to  indicate  specific  performance 
standards  for  training  purjiosos  for  each  terminal  or  facilitating  objective 
on  these  evaluation  plans  f sec  I'igure  5). 


The  evaluation  plans  only  cover  those  areas  of  interest  to  the  school, 
however.  They  usually  do  not  cover  all  the  tasks  in  an  MOS,  especially 
for  the  more  advanced  tasks,  which  the  personnel  are  supposed  to  learn  on 
the  job.  The  Enlisted  Evaluation  Center,  therefore,  has  to  supplement  this 
school-oriented  information  with  other  sources.  Usually,  they  use  the  judg- 
ments of  knowledgeable  NCOs  who  have  experience  in  the  particular  MOS,  and  who 
formulate  proposals  regarding  what  tasks  are  appropriately  included  in  an  MOS 
proficiency  test. 

1.  Criteria  for  the  Training  Objective  developed  for  the  task: 

Action  Troubleshoot  AN/TRC-24.  (A-22-1) 

Condition  In  addition  to  Standard  Training  Conditions,  the 
student  is  given  an  /\N/TRC-24  with  one  major  com- 
ponent containing  a DS  part  defect  as  well  as  OS-8, 
ME-30/U,  TM  11-5820-287-12,  TM  11-5820-287-34,  and 
AN/TRC-24  Block  Diagram. 

Standard  The  student  is  qualified  if,  when  given  two  defec- 
tive AN/TRC-24S  with  a 2-hour  time  limit  on  each, 
he  can  isolate  one  of  the  defective  parts. 


Figure  5.  Evaluation  Planning  Information  Sheet. 


The  proficiency  testing  system  in  the  Army,  as  it  is  presently  organized, 
provides  separate  tests  for  each  skill  level  within  each  MOS.  Occasionally, 
one  test  may  be  used  in  two  or  three  skill  levels  with  different  score  re- 
quirements, but  the  principle  is  the  same.  One  reason  for  providing  separate 
tests  for  different  skill  levels  is  that  an  MOS  is  a broad  collection  of  duty 
areas  that  cover  many  different  duty  positions.  Providing  separate  tests  for 
each  skill  level  makes  it  possible  to  provide  items  that  are  more  appropriate 
for  the  positions  being  filled  by  those  who  take  the  tests. 

This  skill  level  approach,  which  is  not  incompatible  with  the  duty  module 
concept  that  we  have  described,  is  appropriately  used  in  connection  with 
duty  modules,  and,  further,  it  illustrates  how  useful  duty  modules  can  be  to 
those  who  design  tests.  Table  6 lists  the  tasks  for  a duty  module  of  patrolling, 
either  mounted  or  dismounted.  One's  skill  level  is  dependent  upon  whether  one 
supervises  a task,  does  the  task  and  also  supervises  it,  simply  does  the  task, 
or  wliether  one  just  assists  in  doing  it.  The  three  skill  levels  in  the  IIB  MOS 
(Light  Weapons  Infantryman)  are  indicated.  The  skill  levels  are  numbered  1,  2, 
and  4.  There  are  only  three  skill  levels  in  this  particular  MOS,  so  there  is 
no  skill  level  numbered  3.  Table  6 indicates  that  people  who  are  at  skill  level 
4 are  more  likely  to  supervise.  People  who  are  at  skill  level  2 do  not  super- 
vise, and  are  much  more  likely  to  assist  somebody.  People  who  are  at  skill 
level  1 carry  out  the  orders  and  requirements  of  their  superiors.  This  kind  of 
information  about  skill  level  profiles  could  be  extremely  useful  to  Army 
organizations  in  designing  proficiency  tests. 
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Table  6.  Duty  Module  E-5:  Patrols,  Either  Mounted  or  Dismounted 


Performance  Expectations 
Skill  Levels  1,  2 and  4 of 

for 

IIB  MOS‘ 

Tasks 

Super- 

vise 

Do  and 
Supervise 

Do 

Assist 

(1) 

Plan  patrol  operations 

4 

2 

C2) 

Assemble,  inspect,  issue  patrol 
order,  and  lead  patrol 

4 

2 

(3) 

Operate  listening  or 
observation  post 

4 

1 

(4) 

Serve  in  combat  patrols 

4 

2.  1 

(5) 

Serve  in  reconnaissance  patrols 

4 

2,  1 

(6) 

Serve  in  ambush  patrols 

4 

2,  1 

(7) 

Mark  route  or  serve 
as  guide  for  unit'^ 

2,  1 

C8) 

Participate  in  air  search 
operations  or  air  delivered 
patrol 

4 

2,  1 

(9) 

Estimate  charge,  emplace  and 
fire  demolitions 

4 

2.  1 

2^ 

There  are  only  three  skill  levels  in  the  IIB  MOS;  1,  2 and  4. 

^Task  No.  7 (when  performed  by  an  IIB  MOS)  is  normally  supervised  by 
someone  in  another  MOS. 


ADVANTAGE  OF  A MODULAR  APPROACH  IN  TEST  DEVELOPMENT 


Previously  mentioned  were  the  ways  in  which  the  Army  military  occu- 
pational specialties  were  modular.  Refer  to  Table  4 and  note  that  with 
this  matrix  one  can  account  for  many  different  tasks  with  a relatively 
small  number  of  duty  modules.  Moreover,  this  matrix  is  just  the  corner  of 
a much  bigger  matrix.  To  date,  31  enlisted  duty  modules  have  been  developed; 
they  completely  account  for  16  different  3-digit  enlisted  MOSs. 
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To  appreciate  the  possible  savings  it  is  necessary  to  translate  this 
information  on  duty  modules  and  MOS  into  test  items.  Say  that  these  16 
MOS  would  require  100  items  apiece  to  account  for  them  if  they  were  developed 
independently  by  various  test-developing  agencies.  That  would  make  1,600 
test  items,  if  you  used  an  independent  approach  to  test  development.  We 
estimate  that  it  takes  only  10  items  apiece  to  describe  a duty  module. 

In  other  words,  it  is  possible  that  310  items  can  do  essentially  the  same 
job  as  1,600  test  items.  To  be  able  to  prepare  310  items  rather  than 
1,600  items  reflects  a considerable  savings.  It  is  contingent  upon  de- 
fining modules  that  cut  across,  and  have  the  same  meaning  in,  different 
occupational  specialties.  In  giving  these  figures,  we  have  not  discussed 
skill  levels;  but  different  tests  for  different  skill  levels  would  be  re- 
quired in  both  systems.  Thus,  multiply  the  number  of  items  in  our  ex- 
ample by  three  or  four  to  get  the  number  of  test  items  that  would  actually 
be  needed  by  the  people  who  design  these  tests  and  work  with  them. 


MODULAR  ARMY  TRAINING  TESTS  (ATTS) 


At  the  present  time,  the  Army  has  several  hundred  Army  Training  Tests 
(ATTs)  for  use  in  evaluating  the  performance  of  organizational  units. 

Each  of  these  tests  has  a scenario  and  provisions  for  referees  who  are 
trained  to  follow  people  and  take  notes  as  regards  their  performance  in 
the  unit  test.  The  question  to  be  posed  here  is  this:  Would  it  be  de- 

sirable for  Army  Training  Tests  for  organizational  units  to  be  organized 
in  the  same  modular  fashion  that  has  been  proposed  for  individual  pro- 
ficiency tests? 


PURPOSE  OF  ORGANIZATIONAL  UNITS 


Before  further  discussion  of  the  feasibility  of  modular  ATTs,  the 
accuracy  and  consistency  with  which  the  intended  purposes  of  organizational 
units  have  been  specified  must  be  considered.  The  Army  has  several  differ- 
ent terms  for  describing  the  intended  purposes  of  organizational  units: 
a primary  mission,  some  functions,  and  a capability. 

A primary  mission  is  defined  as  the  principal  purpose  that  an  organi- 
zation is  designed  to  accomplish.  The  functions  are  the  appropriate  or 
assigned  duties,  responsibilities,  missions,  or  tasks  of  an  individual 
office  or  organization.  A capability  is  the  ability  to  execute  a speci- 
fied course  of  action.  Further  details  will  not  be  discussed  except  to 
state  that  after  studying  the  various  mission,  functional,  and  capability 
statements  in  Army  documents,  it  was  concluded  that  the  capability  state- 
ment was  the  one  that  should  be  used  as  the  basis  for  structuring  organi- 
zational unit  testing  modules. 


THE  CAPABILITY  STATEMENT 


A possible  capability  statement  for  a theoretical  unit  is  shown  in 
Table  7.  It  is  clearly  possible  to  analyze  a capability  in  terms  of  spe- 
cific component  functions  and  operational  criteria,  as  shown  in  the  table. 


Table  7.  Theoretical  Capability  of  a Unit 


1.  Title: 

Transport  supplies  and  resupply  itself. 

2.  Essential  Component  Functions: 

a.  Load,  move,  and  unload  unit  loads  of  rations, 
POL,  ammo,  and  repair  parts. 

b.  Repair  minor  vehicular  failures  enroute. 

c.  If  unable  to  make  minor  vehicular  repairs,  tow 
inoperable  vehicles. 

d.  Move  unit  loads  on  the  road  or  cross-country. 

e.  Pick  up  and  issue  supplies. 

3.  Minimum  Operational  Criteria: 

a.  Sufficient  vehicles  on  hand  in  condition  to 
move  unit  load. 

b.  Sufficient  vehicles  on  hand  in  condition  to 
pick  up  and  deliver  supplies. 

c.  Trained  drivers. 

d.  Authorized  maps  and  compasses  on  hand. 

e.  Supply  platoon  trained  as  a team. 

f.  Trained  supply  personnel. 

g.  Trained  vehicular  and  radio  mechanics. 

h.  Satisfactory  status  of  equipment  maintenance. 

i.  Satisfactory  completion  of  ATT  and  FTX. 

4.  Standards  (To  be  developed): 

Includes  minimum  personnel,  skills,  operable 
equipment,  and  training  necessary  to  be  con- 
sidered C-1,  C-2,  or  C-3  as  defined  in  AR  220-1. 
Standards  below  C-3  are  C-4. 


The  capability  of  the  unit  is  to  transport  supplies  and  to  resupply  it- 
self. What  is  involved  in  this  capability  are  the  functions  of  loading, 
repairing  vehicles,  picking  up  initial  supplies,  and  so  forth.  The  mini- 
mum operational  criteria  are  also  indicated.  A Department  of  the  Army 
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study  of  output  measurement  conducted  in  1968  shows  the  same  conclusion 
that  is  arrived  at  here--that  capabilities  are  a good  way  to  structure  and 
organize  ATTs. 

CRITERIA  FOR  MODULAR  ATT  EVALUATION  DEVICES 

A number  of  criteria  for  an  improved  system  of  evaluating  performance  in 
Army  Training  Tests  have  been  formulated.  Details  are  beyond  the  scope  of  this 
presentation,  but  the  criteria  and  related  recommendations  are  given. 

* A series  of  clear,  quantitative  statements  specifying  the 
capabilities  of  a unit. 

* A taxonomy  of  unit  capabilit)'  statements.  If  possible, 
these  statements  should  be  modular. 

* Criteria  of  unit  effectiveness  based  upon  and  relatable 

to  the  taxonomy  of  unit  capabilities,  and  consistent  with 
and  relatable  to  criteria  for  the  performance  of  individuals 
in  the  unit . 

* Both  kinds  of  criteria  (individual  and  unit)  based  upon 
performance  standards  rather  than  relative  standing  in  test 
performance. 

* Vatied  performance  standards  depending  upon  situational 
conditions,  such  as  terrain,  percent  casualties,  and 
resource  inputs. 

* Aggregation  of  standards  for  organizational  components  and 
generation  of  an  overall  index  for  the  unit  as  a whole. 

* The  emphasis  in  criteria  statements  upon  end  results  rather 
than  methods  used  to  achieve  the  results. 

* Output  measures  allowing  for  the  possibility  in  evaluations 
of  corrective  measures  that  may  have  been  taken  by  command 
personnel,  and  that  would  permit  the  unit  to  meet  standards 
in  spite  of  some  departure  from  expected  procedures. 

* Scoring  that  provides  specific  information  about  the  leader- 
ship of  a unit. 


Department  of  the  Army.  Improvement  of  Output  Measurement , Report  of  a 
Special  Study  by  the  Army  Staff  Coordinated  by  the  Comptroller  of  the 
Army,  January  1968. 
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* Scoring  that  provides  comprehensive  evaluations  of  the 
unit  when  tested  as  an  entire  unit. 

* Quantitative  weights  assigned  for  satisfactory  performance 
and  deducted  for  inadequate  performance  that  reflect  the 
probable  seriousness  of  the  actions. 

* A method  for  relating  output  measures  to  input  measures 
logically. 

SUMMARY 

A series  of  clear,  quantitative  statements  are  needed  to  specify 
the  capabilities  of  a unit.  There  is  also  a need  for  a taxonomy  of  unit 
capability  statements.  A taxonom>'  is  a theoretically-based  language 
that  implicitly  classifies  or  categorizes  a capability  statement  at  the 
same  time  that  it  describes  it. 

Capabilities  should  also  be  modular  in  nature;  that  is,  the  capa- 
bility statement  for  an  Infantry  battalion  should  be  the  same,  if  pos- 
sible, as  a highly  similar,  closely  related  capability  statement  for  an 
Armored  Cavalry  squadron.  If  you  can  design  the  capabilit)’  statements 
thusly,  and  organize  the  testing  accordingly,  it  is  possible  to  design 
one  modular  unit  training  test  component  that  would  be  useful  for  In- 
fantry batt.llions  and  for  Armored  Cavalry  squadrons.  Examples  of  possi- 
ble modules  are  night  ground  attack,  retrograde  movement,  and  stationary- 
defense.  Since  many  different  capabilities  of  different  kinds  of  bat- 
talions are  common,  having  similar  tests  provide  many  economic  features. 


CURRENT  WORK  ON  MODULAR  ATTS 


Having  proposed  the  design  of  modular  Army  Training  Tests  and  proceed- 
ing to  design  some,  field  survey  work  was  conducted  in  conjunction  with  Army- 
Training  Tests  at  Fort  Lewis,  Washington,  in  August  197.T,  to  pre-test  the 
first  versions  of  these  modular  evaluation  devices.  These  devices  and  data 
have  not  been  analyzed  or  evaluated  as  yet,  thus  comments  are  based  upon  what 
was  learned  during  the  design  stages. 


PERSONNEL  CAPABILITIES 

In  designing  these  evaluation  devices,  two  things  quickly  become 
apparent:  (a)  devices  could  not  be  designed  for  units  based  upon  enlisted 

duty  modules  along;  and  (b)  there  was  a need  for  officer  duty  modules.  A 
contract  already  underway  for  field  job  analysis  of  about  200  officer 
jobs  was  used  as  the  basis  for  designing  initial  duty  modules  for  officer 
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jobs.  Table  8 defines  officer  duty  modules  by  area  and  module  number. 
Table  9 accounts  for  all  the  major  duties  of  an  officer's  job  with  just 
a few  duty  modules.  Each  of  the  positions  shown  is  completely  accounted 
for  by  the  officer  duty  modules  listed.  (The  letter  refers  to  the 
"areas"  from  which  the  module  was  taken  in  Table  8,  and  the  number 
identifies  a specific  module  within  that  area.) 


Table  8.  Officer  Duty  Modules  by  Area 

Number  of 

Area  Title  Modules 

A Command  Management,  General  Management 

and  Administration  9 

B Personnel  4 

C Intelligence  5 

D Operations  and  Plans  (Staff)  4 

E Organization,  Training  3 

F Logistics  (Staff  and  Consumer  Units)  9 

G Communications  and  Electronics  2 

U Civil-Military  Affairs  3 

1 Lomptrollership,  Budget  and  Fiscal  2 

J Army  Aviation  5 

K Research,  Development,  Test  and  Evaluation  2 

L Operations  Research  anj  Systems  Analysis  1 

M ADP  Management  and  Programming  1 

N Education,  Instruction  2 

0 Information  Activities  1 

U Tactical  Direction  of  Combat  Units  5 

W Miscellaneous  9 

X Individual  Functions  and  Special  Qualifiers  4 

FF  Logistical  Services  9 

nil  Supply  and  Maintenance  Support  Operations  9 
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Table  9.  Application  of  Duty  Modules  to  Officer  Positions 


Position 


Duty  Modules 


Cdr. , Infantry  Rifle  Co. 

CPT 

Cdr.,  Reception  Station  Co. 
LT 

Asst.  Army  Attache 
LTC 


A-1, 

A-3, 

A-4, 

A-6 

E-1, 

F-1, 

X-1, 

X-2 

A-1, 

A-3 , 

A-5, 

F-1 

A-1, 

A-4, 

C-4 

EQUIPMENT  CAPABILITIES 

In  addition  to  information  on  the  personnel  capabilities  of  a unit, 
there  was  a need  for  detailed  information  about  the  capabilities  of 
equipment.  The  need  to  think  about  equipment  capabilities  became  ap- 
parent when  preparing  a unit  capability  table,  which  described  different 
types  of  capability  for  each  component  in  a platoon  (Table  10). 


Table  10.  Equipment  Capability  Table  for  Armored 
Cavalry  Platoon  (TOE  17-107H) 


Item  of 

Equipment  Basis  of  Issue  Capability  Reference 


1.  Antenna  (AT-  2-Scout  Section 

784/PRC)  1-Rifle  Squad 


Determine  the  direction 
to  a specific  radio 
transmitting  in  the 
frequency  range  30.0 
to  75.95  MUz. 


ST-24-18-1 


2.  Armored  Re- 
connais- 
sance 
Airborne 
Assault 
Vehicle 
(M  551) 


3-Light  Armor 
Section 


Negotiate  almost  any  ST-17-1-1; 

terrain  at  speeds  ST-17-15-1 

from  4 miles  per  hour  FM-17-36 
in  water  to  43  mph  on 
roads,  including  7- 
foot  spans,  33-inch 
vertical  obstacles 
and  60%  grades . 
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It  was  not  possible  to  describe  the  fire-power  capabilities  of  a 
squad  that  was  equipped  with  a certain  type  of  machine  gun,  for  example, 
without  knowing  what  type  of  machine  gun  it  was.  One  machine  gun  might  be 
capable  of  a sustained  firing  of  40  rounds  per  minute,  while  another 
might  fire  100  rounds  per  minute.  The  range  of  the  first  machine  gun 
might  be  6,000  meters,  while  the  range  of  the  second,  faster  machine  gun 
might  be  3,000  meters.  Clearly,  any  quantitative  statement  of  the  cap- 
abilities of  that  squad,  and  hence  the  capabilities  of  the  whole  platoon,  is 
greatly  affected  by  which  of  the  two  machine  guns  is  being  used.  Similar  con- 
straints upon  unit  capability  statements  are  imposed  by  the  kind  of  trans- 
portation that  is  available.  This  type  of  information  has  obvious  implica- 
tions as  regards  the  capabilities  of  a unit,  and  must  be  incorporated  or 
delineated  in  unit  capability  statements. 


PRELIMINARY  WORK  ON  EVALUATION  DEVICES  FOR  ATTS 


As  stated  earlier,  one  of  the  requirements  for  the  individual  enlisted 
modules  is  that  they  be  meaningful  in  terms  of  unit  evaluation  procedures 
as  well  as  individual  evaluation  procedures.  However,  caution  must  be 
exercised  in  the  using  or  modifying  duty  modules  at  this  stage.  The  task  list 
for  a given  MOS  may  or  may  not  adequately  sample  tlie  tasks  performed  b>' 
incumbents.  The  task  list  also  may  or  may  not  include  all  the  tasks  which 
make  up  the  duty  modules  for  that  MOS.  And  finally,  duty  assignments  may  exist 
which  are  inappropriate  to  a duty  position  designation  within  an  MOS. 

APPROACH  TO  THE  PROBLEM 

The  first  approach  was  to  develop  examples  of  criterion  behavior  for 
a particular  duty  module,  but  it  did  not  seem  to  be  workable  because  too 
much  territory  was  covered  with  one  global  rating. 

A new  approach  was  developed.  Since  each  duty  module  is  defined  by 
approximately  ten  task  statements,  it  was  decided  to  design  the  evaluation 
devices  in  terms  of  checklist  items  based  upon  these  task  statements  CFigure  6). 
The  method  of  grouping  the  items  was  retained  so  that  the  devices  were  still 
organized  in  terms  of  duty  modules.  The  overall  performance  of  the  unit  could 
not  be  scored  exclusively  in  terms  of  individual  performance,  however.  Some 
unit  performance  indices  were  developed.  An  example  of  overall  unit  performance 
rating  procedures  is  sliown  in  Figure  7. 
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1.  Replacements  properly 
received  and  assigned 

2.  Losses  and  casualties 
properly  processed 

3.  Leadership  of  platoon 
and  squad  NCOs 

4.  Duties  of  subordinates 
properly  allocated 


Figure  6.  An  Approach  to  Unit  Evaluation  Devices 
Based  upon  Task  Statements. 


Activity  Rated;  Rifle  Platoon 


Phase  1--Daylight  Attack 


Proper  actions  and  preparations  in 
assembly  area? 

Proper  organization,  formation,  and 
dispersal? 

Platoon's  use  of  cover  and  concealment? 

Firing  on  objective--good  volume,  well 
directed?  (Scored  only  for  live  firing) 


Figure  7.  Evaluation  in  Terms  of  Unit  Performance  As  a Whole 


/ 


Overall  Rating 


RECOMMENDED  APPROACHES  AND  PLANS 


APPROACH 

Tests  have  usually  been  designed  by  taking  a particular  criterion  situ- 
ation and  designing  a test  for  that  one  situation  irrespective  of  how  it  is 
structured  for  that  particular  case  at  that  particular  point  in  time.  If, 
for  example,  a proficiency  knowledge  test  is  being  designed  for  somebody 
who  repairs  automobiles,  you  determine  exactly  which  tasks  people  in  that 
MOS  are  supposed  to  perform,  then  you  design  the  individual  proficiency  tests 
to  measure  knowledge  of  those  specific  tasks. 

The  same  is  true  of  unit  training  tests.  You  look  at  the  capability 
statements,  study  the  terrain  in  which  the  test  is  to  be  conducted,  and 
then  you  design  some  tests  for  that  particular  type  of  unit  in  that  par- 
ticular situation.  If  you  are  dealing  with  an  Infantry  unit,  you  design 
an  Infantry  test;  if  you  are  dealing  with  an  Armored  Cavalry  unit,  you 
design  an  Armored  Cavalry  test.  There  is  relatively  little  overlap  or 
systematic  utilization  of  materials  prepared  by  other  people  for  differ- 
ent kinds  of  units. 

IVhat  is  being  proposed  for  individual  testing  is  that  personnel  experts 
in  the  various  services  spend  a lot  more  time  dealing  with  a variety  of 
occupational  specialties  ratlier  than  focusing  upon  one  specialty  at  a time. 

If  possible,,  they  should  change  the  way  in  which  occupational  specialties 
are  defined,  and  define  different  occupational  specialties  in  terms  of  a 
common  set  of  duty  modules. 

IVhat  is  being  proposed  for  organizational  units  is  that  evaluation 
devices  be  designed  so  that  different  kinds  of  organizational  units  can 
be  evaluated  with  the  same  kind  of  modular  evaluation  devices.  The  modular 
approach,  in  both  cases,  facilitates  feedback  regarding  performance  and 
makes  the  feedback  to  the  individual  or  unit  being  tested  much  more  mean- 
ingful. Both  strong  points  and  weak  points  are  much  better  defined. 

It  is  possible  to  argue  that  feedback  about  test  performance  should 
not  be  too  specific.  The  Air  Force,  for  example,  does  not  provide  infor- 
mation about  area  scores  in  its  Skill  and  Knowledge  Tests  (SKTs) , which 
are  its  equivalent  of  the  Army's  proficiency  test.  The  reason  given  is 
that  people  would  study  for  the  test  rather  than  study  for  the  job  (i.e., 
the  Air  Force  would  seem  to  be  worried  about  the  unintended  consequences 
of  providing  information  in  terms  of  proficiency  testing  area  scores) . 
Available  data  suggest  tliat  area  scores  have  been  successful  in  the  Army, 
however,  and  the  unintended  consequences  simply  have  not  occurred. 

It  would  seem  that,  regardless  of  whether  you  provide  feedback  in  terms 
of  the  modular  scores  that  have  been  described,  or  whether  you  provide  feed- 
back in  terms  of  areas  defined  by  a committeed  of  NCOs  or  unit  commanders, 
area  scores  are  useful  feedback  and  should  be  used. 
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The  Army  has  already  embarked  upon  a large-scale  program  to  overhaul  all 
of  its  unit  training  tests.  The  Army  recently  designed  a systems  engineering 
of  unit  training  programs  that  is  patterned  after  its  systems  engineering  of 
training  procedures  for  individual  training.  Several  organizations  are 
currently  working  on  the  redesign  of  unit  training  programs  in  terms  of  the 
Army's  systems  engineering  of  unit  training  programs.  Thus,  a number  of  new 
evaluation  devices  will  definitely  be  designed.  These  new  approaches  to 
modular  unit  training  tests  have  been  discussed  with  those  who  are  responsible 
for  this  type  of  work,  and  it  is  believed  that  they  will  consider  a modular 
approach  to  ATTs  as  an  alternative  when  the  tests  are  revised. 


USES  OF  DUTY  MODULES 

This  paper  has  stressed  the  economy  of  duty  modules  in  terms  of  test 
preparation  costs  and  the  importance  of  consistency  in  language.  Other 
possible  advantages  are  given. 

(1)  Duty  modules  can  improve  occupational  research,  its 
description,  and  utilization. 

(2)  They  have  the  potential  to  reduce  training  time  and 
lower  training  costs. 

(3)  They  can  provide  a better  use  of  individuals  in 
assignment  substitutions. 

(4)  They  can  simplify  automated  assignment  and  control 
procedures . 

(5)  They  can  improve  proficiency  evaluation. 

(6)  They  can  improve  career  guidance  and  planning. 

(71  They  can  improve  utilization  of  personnel  at  a 
local  level. 

(8)  They  can  improve  unit  training  evaluations. 

PRESENT  IMPLEMENTATION  OF  PROCEDURES 

It  would  take  many  years  before  this  kind  of  modular  system  could  be 
established.  If  everyone  agreed  that  it  should  be  done  tomorrow,  it  would 
still  take  several  years  before  such  a system  could  become  operational. 
Nevertheless,  the  advantages  of  a modular  approach  to  test  construction  are 
so  great  that  people  should  start  thinking  about  it  now.  The  problem  arises 
because  test  developers  cannot  proceed  by  themselves.  The  whole  occupational 
structure  needs  to  be  revised  so  that  occupational  specialties  would  be 
defined  in  terms  of  duty  modules. 
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Nevertheless,  there  are  several  procedures  that  could  be  started  now. 
For  example,  change  the  way  in  which  test  design  work  groups  are  established. 
Instead  of  assigning  test  development  to  a group  of  experts  in  the  same 
specialty,  create  a work  group  comprising  representatives  from  several 
specialties,  and  ask  that  group  to  design  a single  test  component  that  would 
be  useful  to  them  all.  A similar  approach  (i.e,,  assigning  groups  of  experts 
from  diffeient  types  of  units)  can  be  used  with  those  who  design  unit  evalua- 
tion devices.  These  preliminary  steps,  taken  now,  would  have  many  immediate 
advantages,  and  would  greatly  facilitate  a conversion  to  modular  testing  in 
the  future. 
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THE  GROWING  DEMAND  FOR  HUMAN  PERFORMANCE  TESTING 

--  J.  E.  GERBER,  JR.  -- 
ABSTRACT 

This  paper  reports  the  human  factors  involved  in  moving  United  States 
Army  Infantry  School  instructors  away  from  norm -referenced,  know- 
ledge sampling  testing  and  into  criterion-referenced  performance 
testing.  It  sketches  the  gradual  convergence  of  and  increased  com- 
munication between  subject  matter  experts  (instructors)  and  test 
psychologists  (quality  controllers)  from  group  orientation  through 
stylized  example  to  specific  problems  of  performance  testing,  and 
compares  the  dynamics  with  those  observed  by  educators  in  civilian 
schools. 

INTRODUCTION 

Parallel  innovations  have  been  occurring  and  recurring  in  different 
types  of  schools  in  separated  parts  of  the  country. 

In  April,  197  3,  Doll,  Love  and  Levin  (1)  reported  as  follows  on  their 
experiences  in  implementing  a new  instructional  model  in  Louisville 
elementary  and  high  schools. 

"In  Louisville,  Kentucky,  a brave  new  world  of  educational 
innovation  was  recently  attempted.  Bright  ideas  and  good 
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intentions,  a situation  ripe  for  change,  and  a golly-gee, 
whiz -bang  attitude  at  many  levels  of  decision  making 
augured  well  for  its  success.  Education  was  to  roll 
forward.  History  was  to  be  made.  But  somewhere  along 
the  way  the  wheel  had  to  be  rediscovered  and  history  not 
only  was  made,  it  was  remade.  " 

Two  years  ago,  Kramer  and  Kneisel  (2)  showed  you  a model  for 
course  design  as  it  was  then  being  developed  at  the  US  Army  Infantry 
School,  and  related  to  you  a dynamic  plan  for  imposing  (2a)  this  model 
onto  on-going  courses  of  instruction.  Their  model  is  depicted  on  this 
slide. 

(SLIDE  1 ON) 


SLIDE  1.  SYSTEMS  ENGINEERING  PROCESS 


The  two-headed  arrow  labeled  quality  control  in  the  middle  of  the  slide 
implies  that  quality  control  takes  place  at  every  step  of  the  systems 
engineering  process.  Our  concentration  today  will  be  on  criterion 
testing. 

(SLIDE  OFF) 
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Thus,  we  at  Infantry  School  had  a model  and  an  implementation  plan 
for  training  adults  to  perform  specific  jobs. 

Meanwhile,  Getz  and  others  (3),  who  were  changing  from  a tradition- 
ally based  to  competency  based  programmed  instructional  model  for 
teacher  education  at  Illinois  State  University,  provided  the  following 
admonition  in  early  197  3; 

"While  the  rationale  for  any  program  can  be  developed 
logically,  the  actual  implementation  invariably  creates 
many  problems  that  cannot  be  anticipated.  . . The  results 
are  rew'arding.  but  adjustment  by  staff  and  students  is 
slower  and  more  agonizing  than  most  would  suspect.  " 

PURPOSE 

The  purpose  of  this  paper  is  to  compare  human  experiences  at  the 
Infantry  School  with  those  at  other  institutions  in  changing  instruc- 
tional models;  to  focus  on  the  impact  of  changing  from  norm -referenced 
tests  of  knowledge  to  criterion -referenced  tests  of  abilities;  to  discuss 
some  of  the  dynamics  implied;  and  draw  some  inferences. 

Implementation  of  the  systems  engineering  model  at  the  Infantry  School 
has  been  a tremendous  undertaking,  both  from  the  technical  and  the 
human  side. 
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Systems  engineering  documentation  for  the  first  of  some  22  active 
courses  of  instruction  began  in  June  1968.  We  expect  to  have  the 
final  one  completed  in  June  1974. 

During  this  period  through  1972,  the  em.phasis  was  on  documentation. 
Criterion  testing  was  continued  in  those  areas  where  it  alri'ady  ex- 
isted, that  is; 

Pure  performance  testing,  in  which  students  performed  job  tasks 
against  pass/fail  criteria,  continued  in  tests  of  physical  fitness, 
weapons  firing  qualification,  parachute  jump  qualification,  and  land 
navigation. 

Perform.ance  testing  under  conditions  simulating  job  conditions 
and  against  subjectively  rated  criteria  was  carried  on  in  tactical  map 
problems. 

But,  to  a large  degree,  students  were  graduated  from  our  courses  on 
their  ability  to  answer  multiple  choice  items  which  required  recogni- 
tion, recall,  or  comprehension  of  facts  or  doctrinal  principles,  and 
which  were  graded  "on  the  curve.  " 


On  27  Kebruarv  197  3.  we  determined  to  make  all  tests  performance 


»riented  so  that  all  would  have  the  student  apply  job-required  skills. 
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knowledges,  and  attitudes  to  an  identified  job  task.  While  the  rever- 
berations of  this  decision  were  felt  throughout  the  School,  they  were 
experienced  most  acutely  by  the  instructor  who  relied  most  heavily 
on  multiple  choice  items  testing  r ecognition  or  recall  in  his  area  of 
expertise.  In  order  to  relate  subsequent  events,  observe  the  stylized 
functional  diagrani  of  the  School  organizational  activities  on  this 
slide; 


(SLIDE  2 ON) 


SLIDE  2 STYLIZED  (3KO.AN1ZATION  ^ FUNCTION  DIAOH.YAI 


UNOFFICIAL 


The  Assistant  Commandant  runs  the  School. 


The  Director  of  Instruction  is  the  staff  offic-er  in  charge  of  designing 
and  operating  the  resident  courses.  While  he  does  not  have  direct, 
personal  supervision  over  instructional  dejiartment  people,  he  does, 
nevertheless,  dii-ect  or  approve  course  actions,  evaluate  instruction 
and  develop  Programs  of  Instruction  in  the  name  of  the  Assistant 
Commandant.  The  systems  engineers  and  course  coordinators,  work- 
ing within  the  framework  of  a curriculum  committee,  are  the  principle 
architects  of  course  design. 


\ 
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The  test  psychologists  or  quality  controllers  are  the  principal  guard- 
ians of  the  instructional  model  presented  earlier.  Obviously,  they, 
through  their  actions  and  judgments,  are  focal  points  for  convergence 
of  differences  in  rationale,  procedure,  and  requirements. 

The  instructors,  of  whom  there  are  approximately  1,  050  in  the  School, 
have  multifaceted  roles  under  the  systems  engineering  model.  Lectur- 
ing a class  of  students,  as  he  might  have  done  daily  in  earlier  times,  is 
only  one  of  the  instructor's  time  consuming  activities  today.  He  works 
closely  with  systems  engineers  in  documenting  his  area  of  expertise. 

He  prepares  his  lesson  plans  with  all  that  this  entails.  He  works  with 
quality  controllers  in  his  design  and  construction  of  tests  and  examina- 
tions. In  fact,  his  role  is  not  unlike  that  of  the  Illinois  State  University 
faculty  member  described  by  Getz  and  others  (3a)  and  shown  on  the 
next  slide. 

(SLIDE  2 OFF) 

(SLIDE  3 ON) 

SLIDE  3 THE  FOUR  AREAS  OF  RESPONSIBILITY.  . . 

In  addition  to  those  marked  on  the  slide,  the  USAIS  instructor  has  other 
instructional  duties  which  are  physically  and  mentally  demanding  such 
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as  field  problems  and  weapons  firing.  The  instructor  faces  massive 
problems  of  priority. 

(SLIDE  3 OP"F) 

The  multiple  demands  upon  the  instructor's  time  and  attention  account 
in  part  for  the  spectrum  of  response  to  the  requirement  for  perform- 
ance testing.  I emphasize  the  word  spectrum,  for  a spectral  response 
did  occur,  varying  from.  "Great.  Tm  doing  real  world  performance 
testing  already,  " to  "No  way.  " V\e  applaud  the  former  and  the  many 
who  flocked  to  it  in  short  order,  but  it  was  those  on  the  "No  way"  end 
of  the  sc-ale  who  required  the  m.ost  attention. 

This  paper  deals  with  dynamics  of  change.  If  a given  reaction  pr 
attitude  was  encountered  at  all,  it  is  discussed  here  for  its  dynamics, 
not  for  its  frequency. 


iMETHODOLOGV 

Quality  controllers  conducted  a series  of  free-wheeling  meetings  with 
groups  of  10  to  20  instructors  at  a time  to  surface  problems  and  to 
establish  the  kind  of  dialogue  and  openness  which  Doll  and  his  Louis- 
ville associates  (la)  believe  essential.  The  problems  surfaced  bore  a 
marked  resemblance  in  kind  if  not  in  content  to  those  reported  by  Gross 
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and  his  Harvard  colleagues  (4)  in  their  attempt  to  implement  a modu- 
lar change  in  a small  elementary  school.  These  were: 

1)  Staff  resistance; 

2)  Misconceptions  of  the  model  and  roles  within  it; 

3)  Lack  of  expertise; 

4)  Lack  of  materials  and  resources; 

5)  Incompatibility  with  the  model. 

It  became  apparent  to  us  that  not  only  were  there  some  problems  of 
implementation,  many  of  them  were  problems  that  had  been  raised 
elsewhere  some  years  before  as  will  be  related  in  part  below. 

We  determined  that,  having  uncovered  our  own  choke  points  or  bottle- 
necks, we  should  press  on  while  heeding  the  following  speculation  of 
Doll  and  his  associates  (lb): 

. .the  wheel  does  have  to  be  eternally  re-invented.  . . ; 
unless  you  go  through  every  agonizing  step  yourself,  w hat - 
ever  you  accomplish  ultim.ately  will  be  superficial  and 
unimportant.  . .Each  of  us  must  discover  for  ourselves 
where  it  hurts  the  most,  but  the  experience  of  others  may 
be  helpful  in  learning  how  to  cure  the  wound.  " 


304 


In  this  endeavor,  we  determined  to  work  toward  changing  old  concepts 
and  practices  into  new  ones  using  a proactive -retroactive  transitional 
rationale  described  by  the  present  authoi'  (5)  in  1964.  On  the  way  we 
noted  that  Popham  (6),  at  least  as  early  as  1968,  warned  the  educa- 
tional community  that  many  ai'guments  against  ' stating  instructional 
goals  in  terms  of  measurable  learner  behaviors"  were  being  voiced. 
Popham  felt  that  all  these  arguments  were  invalid,  so,  in  order  to 
refute  them,  he  codified  these  arguments  into  eleven  reasons  why  a 
behavioral  model  was  or  is  said  to  be  improper.  Innovators  at  the 
Infantry  School,  at  Illinois  State  University,  (3)  and  Louisville  Public 
Schools  (1)  encountered  several  of  these  reasons  when  changing  ft  om 
one  instructional  or  testing  model  to  another.  Therefore,  it  seems 
appropriate  to  compare  experiences  and  management  techniques  at 
these  three  schools,  using  the  negative  arguments  as  focal  points. 

Perhaps  others  planning  to  roll  toward  pedagogical  innovations  may 
profit  by  our  re-invention  of  the  wheel. 

Popham 's  negative  reason  number  1 is  shown  on  this  slide. 

(SLIDE  4 ON) 

SLIDE  4 NEOATIVE  REASON  1 TRIVIAL  LEARNER  BEHAVK)RS 


(SLIDE  OFF) 


A comparison  of  experiences  and  management  at  the  three  schools 
which  I have  related  to  Reason  1 is  shown  on  this  slide: 

(SLIDE  5 ON) 

SLIDE  5 REASON  1:  "TRIVIA" 


We  see  that  USAIS  and  ISU  (3b)  encountered  the  problem,  whereas 
Louisville  (Ic)  apparently  had  a more  positive  experience  even  though 
some  of  the  sample  objectives  presented  seem  to  be  taxonomicaiiy  low. 

In  our  experience,  operationalizing  at  the  lowest  level  was  easy.  It 
took  the  form  of,  "At  the  conclusion  of  this  instruction,  you  will  be 
able  to  list,  state,  compare,  etc."  The  student  needed  only  sufficient 
m.ental  capacity  to  recognize,  recall  or  understand  instruction  in  order 
to  pass  tests. 

(SLIDE  OFF) 

The  mental  capacities  and  the  elicited  behaviors  are  at  the  lower  end 
of  Bloom.'s  (7)  taxonomy  of  educational  objectives  illustrated  as  a 
stairstep  or  hierarchy  by  this  slide: 

(SLIDE  6 ON) 

SLIDE  6 HIERARCHY  OF  MENTAL  REQUIREMENTS.  . . 
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This  slide,  which  is  a modification  of  Towne's  (8)  illustration  of 
Bloom's  taxonomy,  compares  mental  skills  with  job  performance. 

Our  difficulty  arose  in  trying  to  raise  the  taxonomic  level  from  the 
first  two  levels --knowledge  and  comprehension- -to  the  applicatory 
level  and  higher.  In  the  case  of  knowledge,  we  want  to  say,  "Given 
a situation  such  as  you  are  likely  to  encounter  in  your  future  job,  you 
will  be  able  to  apply  your  newly  acquired  knowledge  well  enough  to 
analyze  your  situation,  synthesize  all  pertinent  factors,  make  an 
evaluative  judgment,  and  behave  appropriately." 

We  found  solidly  imbedded  defense  of  subject  matters  as  having  intrin- 
sic value- -knowledge  for  knowledge  sake  - -regardless  of  the  job  for 
which  the  student  was  being  trained.  The  position  was  com.m.only 
stated  as,  "The  student  just  needs  to  know  this.  " 

(SLIDE  OFF) 

In  managing  or  changing  this  position,  we  had  some  success  in  face- 
to-face  encounters  by  asking,  "Why?"  Success  came  most  often  when 
the  instructor  could  reply.  "Because,  if  the  student  is  ever  in  BLANK 
situation,  he  would  have  to  do  thus  and  so.  " 

Inithis  way  the  barrier  to  higher  order  operationalizing  was  broken  for 
that  particular  instructor  for  that  particular  bit  of  information.  If  the 
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answei-  had  been  generated  more  affectively  than  cognitively,  the 
technique  succeeded  where  logical  argument  and  cogent  reasoning 
had  failed.  Some  instructors  were  more  concerned  with  the  minutia 
of  goal  achievement  than  with  the  achievement  itself.  It  was  more 
important,  for  example,  for  the  student  to  know  the  fact  that  a hole 
must  be  dug  to  a certain  depth  than  it  was  to  specify  acceptable 
behavior  for  determining  hole  depth.  This  attitude  was  amenable 
to  suggestion  that  a slight  change  of  teaching  strategy  would  present 
both  the  doctrine  and  the  term.inai  behavior. 

Popham's  reason  2 (6b)  is  shown  on  this  slide: 

(SLIDE  7 ON) 

SLIDE  7 REASON  2:  PRESPECIFICATION  OF  EXPLICIT  GOALS.  . . 
(SLIDE  OFF) 

The  comparison  of  the  three  schools  is  shown  on  this  slide. 

(SLIDE  8 ON) 

SLIDES  REASON  2;  "UNEXPECTED  OPPORTUNITIES" 

We  see  that  experiences  among  the  three  schools  differ.  The  differ- 
ences may  stem  from  the  different  frames  of  references,  i.e.  , adult. 
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face-to-face  instruction  at  USAIS;  adult  self-instruction  at  ISU  (3a); 
and  elementary  child  face-to-face  instruction  at  Louisville  (Id).  In 
our  experience,  serendipity  in  the  classroom  is  rather  tightly  con- 
trolled both  administratively  and  by  the  students  themselves.  There 
is  a growing  demand  by  students  to  be  informed  of  the  precise  goals 
of  impending  instruction,  aided  by  administrative  demand  for  precise 
statement  of  both  immediate  and  ultimate  goals.  Novel,  spontaneous 
student  responses  in  class  are  encouraged,  but  the  class  members 
themselves  chasten  the  serendipitous  student.  Thus,  even  though  we 
conduct  experimental  instruction,  by  the  very  nature  of  our  training 
task  we  suppress  serendipity  by  choice  and  relieve  boredom  by  a 
multi -media  approach.  Tests,  obviously,  must  test  the  training  ob- 
jectives, not  the  "old  war  story"  which  might  be  used  to  illustrate 
solution  to  a job  task  dilemma. 

(SLIDE  OFF) 

Popham's  reason  3 (6c)  is  shown  on  this  slide. 

(SLIDE  9 ON) 

SLIDE  9 REASON  3:  BESIDES  PUPIL  BEHAVIOR  CHANGES.  . . 


t 


(SLIDE  OFF) 

Comparisons  of  staff  attitudes  and  behaviors  are  shown  on  this  slide. 
(SLIDE  10  ON) 

SLIDE  10  REASON  3;  "ATTITUDE  CHANGE" 


The  reaction  to  proposed  change  was  spectral  at  ail  three  schools, 
so  we  may  concentrate  discussion  on  the  resistance  or  opposition  end 
of  the  scale. 

Some  members  of  the  USAIS  professional  staff  expressed  themselves 
as  follows:  "We  have  gotten  along  without  job  knowledge  performance 
testing  for  nearly  200  years,  and  we  have  won  many  wars  durin^that 
time;  why,  all  of  a sudden,  do  we  have  to  change  now?" 

Those  instructional  staff  members  who  held  this  or  related  views 
presented  the  greatest  challenge  because,  unlike  the  Louisville  plan 
(le),  our  implementation  plan  provided  no  immediate  escape  for  those 
opposed  to  change.  Moreover,  neither  we  nor  the  Louisville  schools 
(la)  had  the  option  of  closing  down  while  we  sorted  ourselves  out. 
Some  viewed  systems  engineering  and  resultant  requirements  for 
performance  testing  as  change  for  sake  of  change,  others  as  schemes 
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dreamed  up  independently  by  quality  controllers  who,  as  a result, 
sustained  a certain  amount  of  personal  denigration  to  which  Doll  (If) 
also  refers.  Some  viewed  the  changeover  as  direct  attacks  on  their 
areas  of  expertise,  pedagogical  approaches,  or  organizational  divi- 
sion of  labor.  Some  saw  the  change  as  a plan  worthy  of  resistance 
or  subversion,  as  the  Louisville  administrators  foresaw  (Ig). 

Even  so,  these  membei  s of  the  staff  responded  positively  to  three 
kinds  of  suggestive  challenge  in  the  military  setting: 

F’irst  - -Look  at  the  kind  of  testing  you  are  doing  now.  If  your 
life  depended  upon  the  ability  of  one  of  your  students  to  perform  in 
your  area  of  expertise,  would  you  say  he  was  qualified  on  the  basis 
of  the  test  you  give  now? 

Second --If  you  cannot  show  how  your  subject  applies  to  the  gradu- 
ate's job,  why  not  delete  your  subject  from  that  course? 

Third- -If  application  of  the  facts  you  teach  is  taught  by  someone 
else,  why  not  stop  trying  to  test  application  now? --Why  not  get 
together  with  the  other  instruction  and  test  later? 

(SLIDE  OFF) 

Popham's  reason  4 (6c)  is  shown  on  this  slide. 
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(SLIDE  11  ON) 


SLIDE  11  REASON  4:  "MEASURABILITY  IMPLIES  BEHAVIOR.  . . " 
(SLIDE  OFF) 

Comparison  of  the  three  schools  is  shown  on  this  slide. 

(SLIDE  12  ON) 

SLIDE  12  REASON  4:  "DEHUMANIZING" 

The  ISU  experience  (3b)  makes  the  case  for  criterion  testing  of  overt 
behavior  at  the  teacher  college  level. 

The  Louisville  experience  (Ih)  makes  a case  for  contingency  planning 
under  a whole  person  interaction  concept. 

As  previous  remarks  on  Infantry  School  experience  with  Reasons  1 
through  3 imply,  we  did  not  run  head-on  into  cries  of  reductionism, 
robotism,  or  similar  charges  of  dehumanizing  either  instructors  or 
students.  In  fact,  part  of  our  underlying  appeal  for  performance 
oriented  testing  has  been  on  grounds  of  human  interdependence.  We 
asked  for  and  received  imaginative  replies  to  this  generalized  ques- 
tion; "What  is  the  minimum  behavior  you  will  accept  now  from  the 
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student  that  will  convince  you  that  he  can  perform  in  your  area  of 
expertise?"  The  replies  were  translated  into  criterion  statements 
of  desired  behavior  and  hence  into  performance  tests. 

In  the  so  called  "soft  skills"  areas --the  leadership  and  attitude  areas 
addressed  by  the  present  author  in  1972  (9)--subject  area  specialists 
working  to  the  point  of  exhaustion  with  test  psychologists  progressed 
from  a condition  of  no  validating  tests,  through  multiple  choice  know- 
ledge regurgitation  tests,  to  case  study  type  situational  judgment  tests 
of  application  of  principles  and  knowledge.  In  addition,  they  have 
developed  leadership  peer  rating  scales  on  the  premise  that  leadership 
qualities  depend,  at  least  in  some  degree,  upon  the  perceptions  of 
those  who  are  to  be  led.  In  addition,  they  have  developed  a judgmental 
rating  chart  for  race  relations  instructors  to  use  in  judging  fitness 
of  students  to  graduate  as  group  facilitators.  Significantly,  here, 
according  to  Rogers  (10),  one  of  the  appropriate  behaviors  is  keeping 
quiet,  remaining  still,  and  allowing  contentious  group  members  to  re- 
late to  the  group. 

Experiences  at  the  three  schools  seem  to  point  to  the  desirability  of 
stating  performance  objectives  in  advance  and  testing  them  by  some 
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means  other  than  the  objective  type,  norm -referenced  paper  and 
pencil  test. 

(SLIDE  OFF) 

Popham's  reason  5 (6d)  is  shown  on  this  slide. 

(SLIDE  13  ON) 

SLIDE  13  REASON  5:  "IT  IS  SOMEHOW  UNDEMOCRATIC.  . . " 
(SLIDE  OFF) 

Comparison  of  the  thi-ee  schools  is  on  this  slide. 

(SLIDE  14  ON) 

SLIDE  14  . REASON  5;  "UNDEMOCRATIC" 

^ 

This  charge  does  not  appear  to  be  a problem  at  any  of  the  three 

schools  (Ig), 

Our  students  are  encouraged  to  demand  to  know  in  advance  the  objec- 
tive of  an  impending  lesson  or  exercise.  -Most  of  the  students  are  in 
courses  of  instruction  by  choice,  planning  their  futures  around  their 
to-be-acqinred  ability  to  perform  in  a given  military  job  field  upon 
graduation.  They  tell  us,  sometimes  vociferously,  when  an  examina- 
tion fails  to  meet  ttieir  expectations  to  allow  them  to  demonstrate  their 
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ability  to  perform  future  job  tasks.  True,  there  are  some  who  com- 
plain about  having  to  make  decisions  on  examinations,  but  not  many. 
(SLIDE  OFF) 

Popham's  reason  6 (6d)  is  shown  on  this  slide. 

(SLIDE  15  ON) 

SLIDE  15  REASON  6;  "THAT  ISN'T  REALLY  THE  WAY  TEACHINO 
IS.  . . " 

(SLIDE  OFF) 

The  comparison  is  on  this  slide. 

(SLIDE  16  ON) 

SLIDE  16  REASON6:  "REALISM" 


Discussion  of  our  experience  relates  to  that  under  Reason  1,  above. 
Our  staff  and  faculty  was  educated,  in  the  main,  under  a liberal  arts 
system  which  emphasized; 

1.  Acquisition  of  knowledge,  to  be  applied-'perhaps--in  that  great 
world  beyond; 

2.  Independent  research,  in  which  knowledge,  vicariously  gained, 
was  self  satisfying  and  possibly  stimulating  to  yet  additional  research. 
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Relatively  few.  including  those  of  the  United  States  Military  Academy, 
have  been  trained  and  educated  to  perform  a specific  job.  Now, 
suddenly,  these  graduates,  who  are  now  faculty  members,  are  re- 
quired to  write  and  teach  toward  behavioral  objectives  and  to  devise 
tests  requiring  appropriate  student  behavior  at  a given  criterion  level 
under  controlled  conditions. 

They  were  not  taught  like  that. 

Teaching  was  not  like  that. 

Teaching  is  now  like  that. 

And,  as  we  are  seeing,  the  change  from  one  frame  of  reference  to 
the  other  spawns  collisions. 

We  have  managed  the  overall  change  through  successive  approxima- 
tion. Quality  controllers  at  first  urged,  helped,  required  instructors 
to  use  action  verbs  in  their  teaching  or  training  objectives,  and  later 
required  more  and  more  specificity  of  the  conditions  and  standards 
of  performance  to  be  elicited  from  the  student. 

Quality  controllers  are  continuing  to  help  instructors  develop  tests 
of  identified  job  tasks  which  are  as  closely  related  as  possible  to 
job  actions,  conditions  and  standards.  While  some  of  the  tests 
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remain  in  the  multiple  choice  mode,  in  which  the  student  selects  one 
of  the  stated  actions  as  his  solution,  we  are  pulling  away  from  these 
as  we  develop  the  capability,  resources,  strategies,  and  instruments 
to  observe  and  grade  outdoor  and  indoor  situational  performance. 

(SLIDE  OFF) 

Popham's  reason  7 (6e)  is  on  this  slide. 

(SLIDE  17  ON) 

SLIDE  17  REASON  7;  "iN  CERTAIN  SUBJECT  AREAS.  ..  ” 

(SLIDE  OFF) 

The  comparison  is  shown  on  this  slide. 

(SLIDE  18  ON) 

SLIDE  18  REASON  7:  "HUMANITIES” 

Obviously,  all  three  schools  encountered  the  problem  (Id,  3b). 

In  our  early  experience,  an  instructor  would  feel  that  systems  engineer- 
ing and  performance  testing  were  fine  for  everyone  else  except  him, 
because  of  the  nature  of  his  subject  area.  A few  went  to  great  lengths 
to  "prove"  that  performance  testing  their  areas  was  impossible.  Their 
cases  were  strengthened,  spuriously,  by  early  inability  of  test 
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psychologists  who  were  unfamiliar  with  specific  doctrine  and  subject 
areas  to  come  up  with  instant  test  items,  schemes  or  scenarios. 

We  managed  this  by  continuing  to  iterate  performance  testing  as  the 
goal  under  a stance  we  termed  "urgent  evolution,  not  instant  revolu- 
tion"; by  frequent,  open,  and  sometimes  protracted  test  item  develop- 
ment sessions  between  instructors  and  test  specialists.  These  took 
on  a Socratic  aura  in  that  the  testers  continually  asked  questions  of  a 
"could  you  do  this"  type  until  the  instructor  himself  eventually  con- 
structed the  test.  We,  in  the  words  of  Doll  and  his  associates  (li); 
"held  fii'mly  to  the  notions  that  people  support  what 
they  help  create,  that  people  affected  by  change  must  be 
allowed  active  participation  and  sense  of  ownership  in  the 
planning  and  the  conduct  of  the  change.  " 

(SLIDE  OFF) 

We  believe  that  the  approach  was  successful  because  both  the  test 
psychologist  and  the  instructor  gave  freely  of  themselves  in  order 
that  the  instructor  might  succeed.  If  this  sounds  humanistic,  it  is 
so  intended,  for  it  fits  rather  well  with  Buehler  and  Allen's  (1  1)  dis- 
cussion of  Bugental's  humanistic  ethic: 
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"Integral  to  his  scheme  is  his  emphasis  that  the  humanistic.  . 
educator —person  hopes  that  via  his  interventions  and  inter- 
actions he  himself  and  the  individual  with  whom  he  is  inter- 
acting will  emerge  from  that  experience  as  societal  change 
agents  themselves.  " 

This  Socratic  tedium  is  still  going  on,  but.  as  Bugental  would  have  it, 
the  conferences  are  more  and  m.ore  frequently  now  between  instructor 
and  instructor  in  the  questioner  and  responder  roles. 

At  this  point  I should  say  that  some  of  our  best  constructed  paper -and - 
pencil  tests  are  the  military  tactics  planning  and  movement  exercises 
laid  on  terrain  maps.  However,  these  tests  have  no  precise  solutions. 
They  are  graded  subjectively  and  serially  by  a team  of  3 tacticians 
who  use  their  own  judgments  as  to  how  good  a tactical  move  the  student 
made  and  how  good  the  student’s  justification  of  the  move  i.^.  *l''ailure 
rates  are  traditionally  high  even  though  norm -referenced  grading  has 
not  been  completely  eliminated.  Quantification  of  tactical  judgments 
in  order  to  determine  why  this  is  so  may  indeed  prove  a difficult  task. 
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Popham's  reason  8 (6e)  is  shown  on  this  slide. 


(SLIDE  19  ON) 

SLIDE  19  REASON  8:  "WHILE  LOOSE  GENERAL  STATEMENTS  OF 
OBJECTIVES.  . . " 

(SLIDE  OFF) 

The  comparison  is  on  this  slide. 

(SLIDE  20  ON) 

SLIDE  20  REASON  8;  "INNOCUOUS  OBJECTIVES" 


As  skills  in  identifying  and  writing  behavioral  objectives  are  being 
developed,  it  is  simply  axiomatic  that  poor,  weak,  or  innocuous  Qbjec- 
tives  will  be  included  (Id,  3b).  The  prospect  is  allowed  for  in  the  all 
pervasive  quality  control  provision  of  the  systems  model  used  at  all 
three  schools  (1,  3).  Infantry  School  instructors,  and  probably  others 
as  well,  found  that  trying  to  construct  performance  tests  on  innocuous 
or  poorly  constructed  behavioral  objectives  was  essentially  impossible. 
This  impasse  also  held  true  when  constructing  tests  on  objectives  which 
did  not  relate  to  the  job  for  which  the  student  was  being  trained. 


During  conferences  with  an  instructor  who  was  having  great  difficulty 
applying  his  subject  area  content  to  the  course  graduate's  job,  the 
test  psychologist  would  help  the  instructor  define  the  problem  but 
would  never  point  the  finger  or  try  to  dictate  course  oi'  lesson  content. 
As  a'result,  instructors  sometimes  drastically  revised  objectives  and 
instruction  in  order  to  teach  and  test  critical  job  skills. 

(SLIDE  OFF) 

Popham's  reason  9 (6f)  is  on  this  slide. 

(SLIDE  21  ON) 

SLIDE  21  REASON  9:  "MEASURABILITY  IMPLIES  ACCOUNTABILITY 

tf 

— — - - t - ■ 

(SLIDE  OFF) 

The  comparison  is  on  this  next  slide. 

(SLIDE  22  ON) 

SLIDE  22  REASON  9:  "ACCOUNTABILITY" 

The  Louisville  (Ig)  elementary  school  innovation  has  an  accountability 
scheme  somewhat  like  that  of  our  own  adult  training  school  wherein 
the  learners,  the  faculty  members,  the  administrators,  and  the  parents 
or.  in  our  case,  employment  market  controllers  have  a voice  in  kind 
and  standards  of  learner  performance. 
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Illinois  State  (3a)  reported  a fragmented  accountability  framework 
which  is  also  similar  to  ours  as  illustrated  on  previous  slides 
(Slides  2,  3). 

Our  faculty  members  do  not  fear  accountability.  In  fact,  most  demand 
it.  Some  argued  tenaciously  for  the  right  to  give  tests  of  knowledge 
and  comprehension,  saying,  "l  want  to  know  that  the  student  knows 
what  I taught  him,.  If  he  fails  the  course  later,  it  will  not  be  because 
I didn't  teach  him  my  subject.  " 

This  stance  was  understandable.  We  managed  it  by  saying  in  effect, 
"You  may  use  whatever  instructional  strategy  you  desire,  including 
tests  of  factual  knowledge.  But  when  the  student  is  certified  in. your 
area  of  expertise,  he  must  be  certified  on  a performance  test.  If 
you  teach  only  facts,  and  someone  else  teaches  application  of  the  facts, 
the  test  must  be  on  the  application.  Result.-  of  your  in-class  informa- 
tion or  practice  tests  can  be  given  to  th^  student's  faculty  advisor  for 
consideration  if  the  student's  progress  appears  in  doubt.  " 

W'e  found  one  ai-ea  which  might  have  been  fear  of  accountability  but  was 
more  likely  a liberal  arts  compassion  for  the  students.  The  instructor- 
proposed  an  examinatioq  of  several  problems  wherein  the  student  could 


r 


omit  one  or  more.  This  plan  was  abandoned  when  test  specialists 
pointed  out  that  it  did  not  assure  testing  of  all  the  training  objectives 
in  the  instructor's  area  of  expertise. 

(SLIDE  OFF) 

Popham's  reason  10  (6g)  is  on  this  slide. 

(SLIDE  23  ON) 

SLIDE  23  REASON  10:  "IT  IS  FAR  MORE  DIFFICULT.  . . " 

[ 

[ 

(SLIDE  OFF) 

I The  comparison  is  on  this  slide. 

(SLIDE  24  ON) 

SLIDE  24  REASON  10:  "GENERATE  OBJECTIVES" 

Whereas  the  ISU  self-instruction  packages  (3b)  are  said  to  contain  all 
the  competencies  considered  necessary  for  teacher  training,  we  noted 

! 

previously  that  not  all  the  higher  order  competencies  were  specified. 

I 

At  Louisville  (Ih),  workshops  in  systems  language  and  in  writing 
behavioral  objectives  were  run  initially  but  were  not  continued  after 
the  first  year.  Instead,  the  school  entered  a plateau  phase--a  time 
during  which  concepts,  ideas,  and  motivations  hopefully  are  being 
internalized. 
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In  our  experience,  some  instructors  raised  the  question  as  to  who 
properly  should  write  the  training  objectives.  Outside  agencies? 
Systems  engineers?  Instructors?  Some  instructors  believed  very 
strongly  that  their  job  was  to  teach,  and  nothing  else.  Some  felt  that, 
because  of  their  junior  officer  grades,  they  were  singularly  unquali- 
fied to  construct  and  ^onduct  tests. 

We  managed  to  gain  their  cooperation  in  helping  to  write  objectives 
by  stressing: 

1.  Need  to  certify  graduates; 

2.  Graduate  capability  to  perform  in  the  instructor's  area  of 
expertise  On  the  job; 

3.  The  logically  inseparable  relation  of  teaching  and  testing; 

4.  Teaching  and  verifying  or  testing  as  two  aspects  of  the  same 
behavioral  modification  process; 

5.  Qualification  to  teach  carries  with  it  the  obligation  to  test; 

6.  Writing  the  action,  conditions  and  standard  of  the  training 
objective  as  tantamount  to  writing  the  test. 

Here  again,  the  approach  was  by  successive  approximations  to  the 


ideal  product. 


(SLIDE  OFF) 


Popham's  reason  11  (6g)  is  on  this  slide. 

(SLIDE  25  ON) 

SLIDE  25  REASON  11;  "IN  EVALUATING  THE  WORTH  OF  INSTRUC- 
TIONAL SCHEMES.  . . " 

(SLIDE  OFF) 

The  comparison  is  on  tiiis  slide. 

(SLIDE  26  ON) 

SLIDE  26  REASON  11:  "UNF'ORESEEN  EVENTS" 


Unforeseen  events  touched  all  three  schools.  Mentioned  earlier  were 
Loui.sville's  (Ih)  underemphasis  on  objectives  for  classroom  manage- 
ment and  IlliiKJis  State's  (3c)  faculty  iind  students'  slow,  agonizing 
adjustment  to  competency-based  teacher  education. 
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as  punishment  and  the  quizzes  as  a whip,  an  outside  motivator  or 
disciplinary  measure  necessary  to  assure  that  students  read  their 
homework  assignments.  These  proponents  would  penalize  students, 
academically,  for  poor  answers  to  pop  quiz  questions. 

The  problem  was  overcome  by  e.xtensive  revision  of  school  testing 
policy  and  by  separating  these  brief  tests  into  two  categories; 

1.  Instructional  techniques,  progress  checks  or  feedback  devices; 

2.  Verifying  techniques,  or  performance  tests  of  ability  to 
perform  job  tasks. 

(SLIDE  OFF) 

SUMMARY 

Hopefully  this  paper  has  answered,  at  least  in  part,  the  following  ques- 
tions about  criterion-referenced  measurement  posed  by  Dziuban  and 
Vickery  (12)  in  February  1973: 

"How  are  teachers  to  make  the  transition  from  the  more 
traditional  practices  and  what  are  the  consequences?" 

"Can  present  instructional  material  be  adapted  to 
criterion-referenced  measurement?  " 


"Will  a new  system  ultimately  result  in  substantial 
additional  demands  upon  teachers,  many  of  whom 
are  presently  operating  on  overcrowded  schedules  ? ” 

CONCLUSIONS 

I believe  that  we  can  draw  the  following  general  conclusion  from  the 
experience  at  the  various  schools  as  presented  and  discussed  here: 
Experiences  with  radical  change  from  one  frame  of  reference  to 
anothei’  have  been  about  the  same  at  USAIS  as  they  have  been  at  other 
schools  that  we  know  about.  In  the  words  of  Doll  and  associates  at 
Louisville  (lb),  this  conclusion  is  stated  as  follows: 

. . problems  which  have  arisen  in  bringing  about 
educational  reforms,  .exemplify  and  reinforce  findings 
of  previous  research.  . . " 


CLOSE 

This  paper  is  not  intended  to  malign  anybody.  Instead,  and  again  1 use 
the  words  of  Doll  and  his  associates  (Ij): 

"it  ...  is  intended  as  a salute  to  people  who  tried  boldly  to 
make  a significant  impact  on  problems  which  bedevil  every 
large  . . . school  system.  Their  mistakes  stand  out  clearly 
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because  they  attempted  more  than  anyone  else  to  date. 
Their  successes  may  seem  unduly  minor  because  they 
attempted  to  conquer  a whole  mountain  range  and  now 
only  control  some  of  the  peaks.  But  they  have  at  least 
gotten  past  the  foothills,  and  from  the  peaks  they  can 
see  more  clearly  than  before.  They  can  continue  to 
climb.  Hopefully,  they  can  help  those  of  us  at  base 
camp  who  may  try  to  join  them.  " 
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CRITERION  REFERENCED  PERFORMANCE  TESTING 
IN  COMBAT  ARMS  SKILLS 

John  F.  Hayes 
URS/Matrix  Company 

My  presentation  today  represents  a progress  report  on  the  work  that 
URS/Matrix  Company  personnel  have  been  doing  in  the  area  of  performance 
proficiency  testing.  Wliile  we  have  been  working  in  this  area  for  over 
four  years,  it  is  specifically  our  current  work  for  the  Army  in  the 
measurement  of  combat  arms  skill  proficiency  that  I want  to  focus  on  today. 

This  work  is  being  done  under  contract  with  the  Army  Research  Institute. 

Dr.  Frank  Harris,  Dr.  Robert  Root,  and  Major  Larry  Word  are  the  ARI 
personnel  monitoring  this  work.  Other  project  personnel  from  URS/Matrix 
include  Mr.  Ray  Griffin,  Dr.  Boyd  Mathers,  and  Mr.  Don  Jones. 

As  background  to  this  description,  I would  like  to  present  a general 
framework  of  performance  testing  that  has  evolved  from  our  work  and  the 
work  of  others  in  this  area.  Performance  testing  itself,  of  course,  is  not 
a new  concept.  It  has  long  been  recognized  as  one  of  the  more  important 
techniques  in  the  area  of  training  assessment.  It  has,  however,  traditionally 
presented  problems  in  terms  of  the  administrative  load  that  it  imposes  on 
the  testing  function  and  in  terms  of  test  reliability.  It  takes  a great 
deal  of  time,  support  materials,  and  it  requires  numerous  personnel  to 
execute.  Still,  it  represents  the  most  valid  means  of  assessing  proficiency. 

This  is  particularly  true  with  respect  to  training  for  specific  job 
assignments  which  is  a major  concern  of  the  military  services. 

Most  general  training  development  models  call  for  the  development  of 
< 

criterion  tests  early  in  the  training  development  process.  Criterion  tests 
developed  from  such  models  are  based  not  upon  the  training  content  being 
developed,  but  upon  the  same  job  requirement  Information  that  was  generated 
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for  the  training  material  development  itself.  The  test  development 
effort,  then,  is  a parallel  and  concurrent  but  independent  effort.  In 
this  way  the  criterion  tests  measure  whether  the  student  can  perform 
criterion  job  functions  upon  completion  of  training,  not  just  whether 
or  not  he  learned  what  was  in  the  course. 

"Criterion  referenced"  tests  as  we  are  employing  the  term  means 
more  than  scores  measured  against  fixed  as  opposed  to  relative  standards. 

By  criterion  referenced  we  also  mean  that  skills  are  measured  in  the 
context  of  the  application  situation  as  completely  as  possible.  By 
criterion  referenced  we  mean  that  the  tests  are  designed  so  that 
performance  on  them  represents  a true  measure  of  job  performance,  rather 
than  measures  of  the  possession  of  skills  presumed  to  contribute 
positively  to  job  performance.  In  the  area  of  Infantry  and  Armor  combat 
skills,  there  are  no  acceptable  job  performance  conditions  against 
which  to  verify  or  validate  evaluation  measures.  In  other  areas  in 
which  we  have  worked,  electronic  and  mechanical  repair  maintenance,  work 
samples  could  be  easily  constructed.  For  the  Infantry  and  Armor  soldier 
this  was  not  the  case.  The  job  of  the  combat  arms  soldier  is  not  to  apply 
a specific  set  of  skills  on  a repetitive  basis.  It  is  to  recall  and  select 
required  skills  as  needed  by  unique  situations  and  integrate  and  apply 
them  under  conditions  of  stress  ranging  from  mild  to  drastically  severe. 
What  we  were  seeking  then  was  to  approach  and  approximate  those  conditions 
in  our  evaluation  of  his  job  proficiency. 

In  our  development  of  criterion  referenced  performance  tests,  several 
guiding  principles  have  become  common  to  all  of  our  developmental  efforts. 
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The  first  is  that  we  start  by  defining  actual  job  tasks.  These  tasks 
come  from  the  analysis  of  the  performance  of  job  incumbents.  This 
information  can  be  gathered  in  a variety  of  ways.  It  can  be  done  through 
conventional  surveys  of  job  requirements  and  job  duties,  or  by  obser- 
vation of  job  incumbents  in  the  performance  of  their  duties,  or  it  can 
be  done  through  the  "panel  of  experts  approach".  The  method  used  has 
to  be  selected  based  on  the  best  source  of  information  available.  Such 
methods  are  not  a central  topic  for  today's  discussion;  however,  a 
fundamental  premise  of  our  test  development  approach  is  that  through  the 
use  of  such  data  validity  is  built  Into  the  test  situation  from  the 
beginning.  The  validity  of  the  performance  test  results,  then,  is  not 
demonstrated  by  comparison  with  other  indices  such  as  supervisor  ratings 
or  other  test  score  results  that  are  more  typically  used  for  test 
validation.  Content  validity  is  established  as  a precondition  so  that 
the  final  tests  represent  absolute  criteria  against  which  any  other 
indices  of  job  performance  can  be  themselves  validated. 

The  second  characteristic  of  our  performance  tests  are  that  meaning- 
ful size  tasks  are  selected,  which  require  skill  integration  and  selection 
from  a total  repertoire  of  application  skills.  In  performing  a job  duty, 
it  is  important  that  the  job  incumbent  be  able  to  handle  all  of  the 
parameters  of  that  job  duty  in  the  context  and  sequence  that  he  will  be 
expected  to  perform  on  the  job.  Under  our  concept,  performance  testing 
is  not  just  the  isolated  performance  of  defined  skills,  but  would  include 
the  selection  of  skills  to  be  applied  at  any  given  time  based  upon  the 
parameters  of  the  situation.  For  example,  a technician  may  be  perfectly 
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capable  of  repairing  a carburetor,  and  this  could  be  tested  specifically. 
It  is  also  important  in  our  view,  that  he  be  able  to  recognized  when  the 
carburetor  needs  to  be  repaired.  That  is,  he  must  be  able  to  isolate 
the  carburetor  as  a malfunctioning  component  from  among  the  myriad  of 
other  engine  components.  Further,  the  performance  of  this  task  should 
be  done  in  the  context  of  the  operational  environment.  To  the  extent 
possible  the  individual  should  have  to  locate  the  problem,  select  tools 
and  equipment,  use  appropriate  job  documentation  effectively,  correct 
the  problem,  and,  in  general,  demonstrate  his  ability  to  operate  within 
the  operational  environment.  This  then,  is  what  we  mean  by  total  skill 
integration. 

The  third  major  characteristic  of  the  performance  oriented  tests  is 
that  the  measures  that  are  used  for  gauging  successful  performance  are 
product-oriented  rather  than  process-oriented.  When  a man  is  functioning 
as  part  of  a system  he  has  inputs  and  outputs  the  same  as  any  other 
element  of  that  system,  either  man  or  machine.  The  functioning  of  the 
system  depends  upon  the  quality  and  timeliness  of  the  man's  outputs. 

These  characteristics  can  be  measured  as  can  the  outputs  of  any  other 
component  of  the  system. 

Some  of  the  more  compelling  aspects  of  performance  for  measurement 
purposes  are  often  processes  and  in  many  instances  the  quality  of  such 
processes  is  often  a matter  of  subjective  judgement  on  the  part  of  a rater 
or  evaluator.  He  will  observe  whether  the  performing  individual  ;.id  the 
task  in  accordance  with  prescribed  procedures  without  questioning  whether 
those  procedures  were  in  fact  the  only  way  of  accomplishing  the  output. 


361 


In  the  design  of  our  performance  tests,  we  have  conscientiously  selected 
criterion  measures  which  do  not  rely  upon  the  subjective  judgement  of  the 
rater.  The  guiding  philosophy  of  this  test  development  effort  was  that  we 
do  not  care  how  a man  accomplishes  his  job  or  his  task.  What  we  are 
concerned  with  is  whether  he  got  the  job  done,  i.e.,  produced  the  required 
outputs  at  the  proper  time.  In  this  way,  the  testing  procedures  can 
remain  independent  of  the  specific  processes  that  are  prescribed. 

Applying  these  principles  to  testing  combat  skills  of  the  Light 
Weapons  Infantryman  (MOS  IIB) , we  began  with  a definition  of  the  situations 
in  which  the  individual  is  expected  to  perform.  These  can  generally  be 
defined  in  terms  such  as:  assault  and  defense,  reconnaissance  patrol, 

defensive  patrol,  night  infiltration,  and  anti-tank  actions.  The  next 
step  was  to  perform  a behavioral  task  analysis,  in  cue-response  terms, 
to  determine  exactly  what  the  individual  soldier  is  responding  to  in  each 
of  these  situations  (the  cues)  and  what  outputs  (response)  are  expected 
of  him. 

One  of  the  first  characteristics  that  became  evident  was  that  while 
we  were  talking  in  terms  of  individual  proficiency  and  individual  testing, 
the  soldier  seldom  acts  as  an  individual  in  the  defined  criterion  situations. 
He  is  virtually  always  a member  of  a larger  unit,  most  often  the  fire 
team,  operating  as  a member  of  a squad.  As  we  proceeded,  it  became  clear 
that  we  had  to  test  him  in  that  context. 

Another  critical  factor  that  became  clear  is  that  the  infantryman  is 
operating  most  of  the  time  against  an  intelligent  opposing  force,  that  is 
only  vaguely  predictable  in  its  actions  and  which  can  kill  him  unless  it 
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is  neutralized.  Initially  this  element  was  considered  too  dynamic  and 
too  difficult  to  provide  to  be  considered  for  inclusion  in  a test.  The 
more  we  examined  the  situation,  however,  the  clearer  it  became  that 
without  this  element,  attempts  to  test  the  combat  efficiency  of  the 
individual  soldier  were  going  to  be  woefully  inadequate.  While  we 

I investigated  hit-indicator  systems  such  as  laser  beams  and  radio  activated 

P. 

n 

; equipment,  we  felt  strongly  that,  to  be  effective  and  usable,  any  system 

that  we  devised  had  to  be  free  of  complicated  equipment. 

Eventually  we  devised  a system  which  uses  a two-digit  number  placed 
on  each  individual's  helmet  and  rifles  equipped  with  telescopes.  Wlien  an 
infantryman  can  distinguish  an  opponent's  number  through  his  scope  and 
fires  a blank  round,  it  is  assumed  that  he  would  have  been  able  to  kill 
that  individual.  By  placing  test  controller  personnel  with  each  of  the 
opposing  elements  and  equipping  the  controllers  with  a radio,  individuals 
can  be  knocked  out  of  action  on  each  side  as  they  are  effectively  engaged 
by  the  opposing  force.  The  problem  is  thus  allowed  to  play  itself  out 
on  a realistic  basis.  By  experimenting  with  combinations  of  color, 
background,  and  size  of  the  numbers  and  the  power  of  the  scope,  a combina- 
tion of  number-color  and  number-size,  and  scope  power,  was  achieved  which 
duplicates  closely  the  effective  range  of  the  M16  rifle.  For  all 
practical  purposes  this  range  is  200  meters.  Uliile  the  actual  range  of 
tl>e  M16  is  much  longer,  the  combat  situation  seldom  provides  accessible 
targets  beyond  that  distance. 

The  developing  of  this  scoring  system  opened  new  possibilities  to 
the  use  of  combat  situations  for  testing  purposes.  The  situations  could 
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now  provide  realistic  cues  which  are  highly  similar  to  the  combat  situation. 
They  also  provide  the  opportunity  for  the  individual  soldier  to  select  and 
employ,  in  concert  with  his  fellow  squad  members,  what  he  considers  to  be 
effective  responses.  The  required  responses  are  not  pre-defined;  he  has 
to  select  them  from  his  repertoire,  under  stressful  conditions,  at  the 
most  appropriate  time,  in  order  to  be  successful.  Further,  the  situation 
will  now  change  as  a result  of  the  Individual  acts  taken  on  both  sides. 

When  the  individual  perceives  that  he  is  affecting  the  outcome,  he  becomes 
a very  interested  and  enthusiastic  participant  in  the  exercise.  When  the 
situation  does  change  as  a result  of  actions  one  takes,  it  becomes  a 
measurement  criteria.  One  of  the  most  important  elements  of  the  infantryman's 
life  is  that  fact  that  he  must  constantly  respond  to  an  environment  that 
is  changing  under  very  stressful  conditions.  And  his  responses  cannot  be 
isolated  in  terms  of  his  own  situations  as  an  individual,  but  must  be 
coordinated  with  his  fellow  squad  members,  if  they  are  to  be  effective. 

Techniques  were  then  developed  for  simulating  and  measuring  the 
effects  of  the  other  weapons  available  to  the  infantryman.  The  tests 
now  permit  the  use  of  grenades,  claymore  mines,  booby  traps,  anti-tank 
weapons,  machine  guns,  and  artillery  in  realistic  fashion.  As  a result, 
the  oveiwhelming  opinion  of  experienced  combat  infantry  personnel  is 
that  we  have,  in  fact,  created  situations  which  call  for  very  realistic 
responses  in  conditions  very  closely  approximating  actual  combat. 

Having  achieved  this,  it  was  then  necessary  to  apply  the  necessary 
controls  to  achieve  individual  testing  within  this  context.  The  basic 
data-gathering  instruments  during  these  situations  are  the  test  controllers 
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previously  mentioned.  In  addition  to  being  equipped  with  radios  in  order 

to  make  the  situations  play  realistically  and  to  assess  casualties, 

control  personnel  have  responsibility  for  recording  product  outputs  for 

each  of  the  individual  participants.  The  controller  keeps  track  of  who 

scored  what  kills  or  was  killed  with  rifle  fire,  who  threw  grenades  or 
was  killed  by  grenades,  who  called  in  artillery  or  was  killed  by  artillery 
fire,  who  employed  booby  traps,  mines  or  claymores,  and  who  was  killed 
by  these  devices.  The  mechanisms  developed  for  keeping  track  of  this 
information  rely  primarily  on  the  use  of  the  identifying  numbers  on  each 
individual's  helmet.  Points  are  accumulated  for  each  Individual  on  the 
basis  of  the  number  of  kills  that  he  has  scored,  plus  a factor  added 
for  the  degree  of  success  achieved  by  this  unit.  The  unit  score  is 
determined  on  the  basis  of  a set  number  of  points  for  achieving  the 
objective,  less  a set  number  of  points  for  each  casualty  incurred.  The 
Individual  scores  are  then  loaded  with  a factor  based  upon  unit  success. 

At  the  present  time,  both  the  defense  and  the  offense  are  being  scored 
simultaneously.  Therefore,  the  degree  of  success  of  one  is  diminished 
by  the  score  of  the  other. 

There  are,  of  course,  many  chance  factors  operating  In  an  individual's 
score,  just  as  in  combat  there  are  many  chance  factors  operating  to  affect 
one's  mortality.  A man  can  get  killed  by  his  own  squad  members,  he  can 
be  killed  by  an  ill-launched  grenade  which  happens  to  bounce  off  a tree 
back  into  his  own  position,  or  by  calling  artillery  fire  on  himself. 

Some  of  these  are  his  errors, some  are  the  errors  of  others.  To  overcome 
these  chance  factors,  repetition  of  testing  is  employed.  Going  through  a 
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test  situation  one  time  will  not  necessarily  produce  an  index  of  the 
best  members  of  a squad,  or  absolute  individual  proficiency  scores. 

Over  the  course  of  a series  of  repetitions  of  the  test,  however,  it  is 
hypothesized  that  proficient  individuals  will  achieve  higher  scores 
than  those  who  are  not  proficient.  That  is,  the  chance  factors  will 
gradually  balance  out  for  all  participants.  Furthermore,  it  is  necessary 
for  the  situation  to  be  run  numerous  times  to  provide  exposure  of  the 
individual  to  different  situations.  Even  running  i.he  same  problem  over 
the  identical  terrain  produces  an  altered  situation.  The  fact  that  no 
two  problems  ever  are  run  alike  with  exactly  the  same  results  or  events 
occurring  is  also  assumed  to  be  a valid  construction  in  terms  of  the 
combat  situation.  Each  combat  situation  is,  in  fact,  unique  and  being 
able  to  respond  properly  to  unique  situations  is  the  highest  measure  of 
a combat  infantryman's  proficiency.  This  then  is  what  we  mean  by  total 
skill  integration. 

To  provide  a better  description  of  the  system  in  operation  I have 
some  slides  taken  during  our  tryout  of  the  Infantry  tests  at  Fort  Benning 
wliich  depict  several  of  the  criterion  test  situations. 

Testing  situations  were  developed  for  each  of  the  other  defined 
criterion  situations  using  the  same  guiding  principles  as  for  the  assault/ 
defense.  The  individual  was  to  receive  the  same  Inputs,  perceive  the  same 
environment  and  make  the  same  response  as  would  be  required  by  the 
criterion  job.  If  there  was  a non-predictable  enemy  present  in  the 
criterion  job  he  was  provided  in  the  test  situation.  In  the  reconnaissance 
patrol,  for  example,  a squad  was  given  an  objective  to  recon,  a map  and  an 
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( operations  order  describing  what  they  might  encounter  in  trying  to  gain 

i 

intelligence  about  the  objective.  The  defensive  patrol  on  the  other  end  | 

was  given  a sector  to  patrol  and  protect  against  recon  elements.  They  | 

! 

could  employ  any  strategy  they  chose,  such  as  a fixed  screen  with  booby 
trap,  a moving  screen,  or  any  combination  thereof.  The  problem  was  then 
started  at  a prearranged  time  and  it  ended  when  the  recon  element 
returned  or  was  neutralized. 

Upon  the  return  of  the  recon  patrol  all  surviving  members  were  tested 
by  having  to  report  wliat  they  saw  at  the  objective.  The  squad  was  awarded 
points  based  upon  the  percentage  of  intelligence  returned  by  any  members. 

The  objective  of  a recon  patrol  is  to  get  accurate  and  complete  intelligence 
about  the  enemy  back,  to  the  base  camp.  If  only  one  man  gets  back  but  he 
has  complete  information,  the  mission  is  a success.  If  all  get  back  but 
with  only  sketchy  or  inaccurate  information  the  mission  is  a failure. 

The  scoring  system,  therefore,  has  to  be  constructed  so  as  to  promote  the 
correct  outcome  and  penalize  incorrect  ones.  One  of  the  easiest  traps  to 
fall  into  in  performance  testing  is  to  construct  a scoring  system  that  is 
^ simple,  objective,  easy  to  use,  and  directly  opposed  to  the  promotion  of 

the  desired  behaviors.  Overloading  the  reward  for  surviving  or  for  correct 
performance  of  enabling  processes  without  due  consideration  of  results, 
is  self-defeating.  The  constant  concern  with  test  and  training  situations 
that  are  this  realistic  must  be  that  they  do  not  teach,  through  design 
or  default,  habits  that  will  get  people  killed. 

The  Infantry  tests  were  field  tested  utilizing  troops  of  the  197th 
Infantry  Brigade  at  Fort  Banning,  Georgia,  during  August  of  this  year. 

i 
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The  experimental  design  plan  called  for  the  total  pool  of  test  items  to 
be  administered  to  a trained  Infantry  platoon  that  could  be  expected  to 
know  and  be  able  to  perform  at  a reasonable  level  of  proficiency.  The 
individual  test  items  that  correlated  best  with  total  test  results  were 
then  to  constitute  the  final  proficiency  test.  Further,  this  tryout  was 
to  determine  the  feasibility  of  the  scheduling  and  times  allotted  for 
each  of  the  test  items. 

As  indicated  earlier,  no  independent  criteria  were  considered  suitable 
as  validation  criteria,  since  no  other  criteria  were  considered  to  be  as 
valid  as  the  tests  themselves.  The  elements  of  the  test  content  were 
reviewed  by  numerous  infantry  experts  and  judged  to  be  valid  samples  of 
the  infantryman's  job.  Therefore,  agreement  or  disagreement  with  indices 
such  as  supervisors'  ratings  or  written  test  scores  was  not  considered 
appropriate.  To  determine  wliether  the  results  did  agree  with  commonly 
accepted  judgements,  peer  ratings  were  employed  which  asked  for  ratings 
of  preference  of  individuals  to  go  into  combat  with. 

Test  Evaluation 

At  this  point  in  time  we  have  only  had  an  opportunity  to  take  a cursory 
look  at  the  results  achieved  during  the  tryout  of  the  Infantry  tests. 

Following  completion  of  the  Armor  test  tryout  more  complete  analyses  will 
be  conducted  of  all  the  data. 

The  Infantry  tests  were  administered  to  one  Infantry  platoon  consisting 
of  35  people.  Nine  were  Squad  and  Fire  Team  Leaders  (designated  as  MOS  11B.40) 
and  26  were  squad  members  (11B.20).  Formal  test  administration  took  one  week. 


Prior  to  test  administration  pre-training  was  conducted  to  insure  that 
personnel  were  familiarized  with  the  proper  employment  of  the  rifle 
scope,  since  that  was  a non-standard  item  of  equipment. 

Several  weeks  were  also  spent  prior  to  the  test  administration  in 
training  control  personnel  (NCO’s)  in  the  details  of  test  administration 
and  in  insuring  that  the  Infantry  personnel  had  sufficient  minimum  skills 
to  participate  in  the  tests.  This  was  necessary  since  the  platoon  we 
were  furnished  was  newly  formed  with  personnel  just  out  of  individual 
training  who  had  no  unit  training  or  experience.  Since  the  tests  were 
designed  for  journeyman  level  infantrymen,  basic  skills  in  map  reading, 
artillery  adjustments,  squad  tactics,  etc.  had  to  be  covered  so  that 
these  personnel  could  be  tested. 

After  the  week  of  testing  the  platoon  members  were  requested  to  rank 
each  other  in  terms  of  whom  they  would  most  prefer  to  have  with  them  in 
combat.  While  they  had  participated  with  an  against  everyone  in  the 
platoon,  no  one  had  any  knowledge  of  the  test  score  results  at  the  time 
of  this  rating.  It  was  based  on  what  the  individuals  had  seen  during  the 
month  of  training  and  testing. 

The  raw  scores  for  the  2 level  personnel  (n=26)  (squad  members)  ranged 
from  75-227  with  a mean  of  214.96  and  a standard  deviation  of  48.26.  For 
4 level  personnel  (n=9)  (Squad  and  Fire  Team  Leaders),  the  range  was 
158-434  with  a mean  raw  score  of  335  and  a standard  deviation  of  96.34. 

The  correlation  of  total  test  score  with  the  peer  ratings,  computed 
by  Pearson  r,  was  .697  for  4 level  personnel  and  .420  for  2 level  personnel. 

Both  are  significant  at  the  05  level  of  confidence. 
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The  Combat  Performance  sub-portion  (two-sided)  of  the  total  test 
correlated  with  the  total  test  score  at  .942  and  .927  respectively  for  the 
2 and  4 level  personnel. 

Due  to  the  built-in  content  validity  of  the  test  situations,  the 
agreement  of  test  results  with  peer  rating  data,  and  the  spread  of  scores 
obtained,  we  are  satisfied  that  these  situational  performance  tests 
are  yielding  an  index  of  individual  combat  proficiency  for  the  Light 
Weapons  Infantryman. 

Armor  Crewman  Tests 

A similar  set  of  criterion  tests  were  developed  for  Armor  crewmen. 

The  duties  of  the  Armor  crew  positions  are  well  defined  and  they  combine 
with  the  tank  to  make  a single  weapons  system.  This  is  both  a simplifying 
and  complicating  characteristic.  The  end  product  of  a tank  gun  round  on 
target  involves  the  coordinated  actions  of  four  people  which  makes  the 
job  of  ferreting  out  individual  contribution  and  proficiency  more 
difficult.  The  proficiency  of  the  tank  crew  as  an  entity  is,  and  has 
been,  a common  level  of  distinction  for  proficiency  measurement. 

To  break  through  to  individual  proficiency  we  went  to  a process 
checklist  type  of  evaluation  within  each  situation  when  it  was  not 
possible  to  get  product  measures  on  each  individual.  The  tank  criterion 
situations  Include  tank-to-tank  engagement,  night  bivouac  situations, 
infiltration,  a river  crossing,  overcoming  mined  obstacles,  bunkers, 
ambushes,  and  maintenance. 
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The  primary  scoring  system  used  in  tank  engagement  is  to  sight  through 
the  main  gun  via  a telescope  as  with  the  infantry  tube  weapons.  This 
provides  immediate  verification  to  the  controller  as  to  whether  a hit 
would  have  been  achieved.  He  can  then  take  the  appropriate  action  to 
kill  the  other  tank  (via  radio)  or  notify  the  crew  that  the  round  missed 
and  the  situation  continues.  Against  enemy  infantry,  the  same  scoring 
system  is  used  as  before  that  of  reporting  helmet  numbers. 

The  Armor  tests  are  currently  undergoing  field  evaluation  at  Fort 
Carson,  Colorado,  and  no  results  are  available  at  this  time. 

Conclusions 

Our  experience  to  date  in  this  project  has  reinforced  our  belief 
that  performance  testing  could  be  employed  to  measure  larger  and  more 
meaningful  segments  of  job  performance  than  has  generally  been  the  case 
to  date.  Certainly  the  control  and  data  collection  problems  increase, 
but  the  increased  validity  of  the  exercise  and  the  resulting  increase  in 
confidence  of  the  results  appears  to  be  worth  the  costs.  The  change  in 
emphasis  is  from  having  the  individual  demonstrate  that  he  knows  or  can 
perform  each  individual  skill  required,  to  having  him  recall  and  effectively 
apply  such  skills  as  the  situation  demands.  While  there  may  be  less  control 
over  which  specific  skills  are  tested,  the  sampling  can  be  handled  through 
both  repetition  and  situation  design. 

These  tests  are  not  yet  in  the  final  format.  Refinement  and  further 
research  is  required  to  improve  the  quality  of  the  data  collected  and  to 
increase  the  discrimination  power  of  the  results.  We  strongly  feel,  however. 


371 


that  these  are  problems  of  technique  and  not  the  inherent  nature  of  perfor- 
mance tests.  The  challenge  is  to  pursue  the  refinement  of  techniques 
rather  than  back  off  to  more  manageable  though  less  fruitful  approaches 
to  testing.  Performance  testing  has  the  power  of  accurate  job  proficiency 
assessment  and  the  challenge  is  to  tap  it. 

As  to  the  increased  costs  in  time  and  manpower,  we  feel  that  a change 
in  attitude  and  commitment  to  the  testing  process  is  needed.  If  we  now 
have  the  beginning  of  mechanisms  for  accurately  determining  the  status 
of  proficiency  of  our  personnel,  the  resources  should  be  made  available 
to  employ  it.  Without  the  type  of  data  that  is  available  from  well  designed 
performance  testing,  training  efforts  will  necessarily  operate  at  less  than 
optimum  efficiency. 
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THE  AIR  FORCE  WIFE:  HER  KNOWLEDGE  OF,  AND 
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This  paper  prepared  for  presentation  at  the  Military  Testing 
Association  Conference  on  "Human  Resources  - Growing  Demands," 
Oct.  28  - Nov.  2,  1973,  San  Antonio,  Texas.  A more  analytic 
treatment  of  the  material  presented  in  the  paper  may  be  found 
in  technical  report  #103,  "The  Air  Force  Wife  - A Study  of  Morale 
among  Military  Dependents"  issued  by  the  Center  for  Human 
Appraisal,  Wichita  State  University. 


*The  Research  reported  in  this  paper  was  supported  by  the  Air  Force 
Office  of  Scientific  Research,  grant  72-2001. 
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A major  corporation  gives  gold  charms  in  the  shape  of  each 
new  state  to  wives  of  transferred  executives  and  it  becomes  status 
to  point  with  pride  at  the  number  of  charms  acquired  during  a career 
span.  A novelty?  Public  relations  gimmick?  No,  rather  a small  part 
of  the  concerted  effort  in  industry  to  make  wives  feel  a part  of  their 
husband's  career  and  satisfied  with  their  place  in  this  future.  The 
industry  motives  are  far  from  altruistic  for  as  William  Whyte  stated 
in  1948  "as  an  econoic  lever ...  companies  have  learned  that  there  is  no 
stimulus  quite  so  effective  as  the  wife  if  proparly  handled."  Much 
research  has  gone  into  ways  to  improve  the  recruiting  and  retention  of 
civian  employees  and  maximize  their  productivity.  Helfrich,  (1965) 
concluded  that  "corporations  are  increasingly  interested  in  the  wives 
of  their  executives."  In  an  1971  study  of  business  executives  J.M.  and 
R.F.  Pahl  typified  the  comments  of  their  interviewees  on  the  single 
most  important  factor  influencing  their  career,  "my  wife,  more  than 
anything  else." 

It  would  logically  follow  then  that  the  military  in  many  ways  would 
not  only  follow  this  pattern  but  demand  more  from  wives  and  family. 

And  likewise,  the  commitment  a man  makes  in  choosing  a military  career 
would  call  for  support  and  approval  from  his  wife  and  family. 

As  our  society  has  become  more  and  more  mobile,  the  extended  family 
of  the  past  has  been  replaced  by  a more  insular  unit.  No  longer  does 
a man  have  relatives  close  by  or  living  in  his  home,  for  his  many  moves 
may  take  his  family  far  from  any  familial  ties.  Without  these  other 
sources  to  rely  on,  the  family  tends  to  turn  to  each  other  and  inter- 
personal relationships  attain  a higher  value  in  assessing  each  individual's 
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satisfaction.  The  demands  of  a military  life  emphasizes  this 
phenomenon . 

One  wife  in  a recent  survey  of  military  families  related  that 
her  eldest  child  had  attended  14  schools  before  entering  college. 
(Entrails,  1971)  How  she  views  this  style  of  life,  the  rewards  it 
offers  her  and  her  family  must  be  weighed  against  the  deficiencies. 

And  the  conclusions  she  reaches  concerning  these  matters  will  play 
an  important  role  in  her  husband's  life  also.  He  has  drawn  his 
family  into  a life  style  unique  in  many  aspects;  a subculture  of 
over  4 million  dependents.  Roger  Little  (1971)  has  remar)ced  on  this, 
"All  m(.litary  families  have  in  common  knowledge  and  experience  in 
an  occupational  culture  ( or  subculture  ) which  is  more  distinct 
than  t.hat  of  other  occupations  in  the  larger  society."  Base  housing, 
required  mobility,  the  status  in  the  community  of  the  military  man  - 
these  are  only  a few  of  the  pressures  a man  choosing  a military  career 
must  weigh.  Add  to  these  the  similar  pressures  his  wife  and  '"imily 
meet  and  one  can  see  either  an  effective  cohesive  unit  or  a point  of 
dissention  for  a man  trying  to  fulfill  a role  as  a husband  and 
father . 

In  a recent  survey  of  career  attitudes  among  Air  Force  personnel, 

it  was  found  that  the  wife  had  twice  the  influence  upon  her  husband's 

career  intentions  as  any  other  individual,  including  his  immediate 

(Belt,  1972) 

supervisor  or  any  of  his  peers. f A determinant  as  important  as  this 
can't  be  slighted  or  ignored.  In  an  investigation  of  junior  officer 
retention  problems,  Lund  (1972)  found  that  wives  were  the  key  variable 
in  the  decision  to  separate  or  remain  in  the  army.  A 1971  study  empha 
that  any  good  junior  officer  retention  program  must  include  efforts 


aimed  toward  the  wife.  "The  military  family  has  come  to  be  considtred 
ar.  essertial  coiiponent  in  f’ersonnel  policy  and  management.”  (lurd, 
1971) 

How  .1  man's  fami  1'/  perceives  him  and  his  role  in  the  community 
exei  Ls  1 1 t,p.i  ndous  imp'Sct  on  his  cwn  view  of  him.self  and  t!.<_  valm 
of  iii£.  woi'r. . This  affects  not  only  his  attitude  but  inci  lasi  i.gi  y 
his  perform.ancc  and  his  rclationship-s  with  those  around  liim.  Kuci. 
effort  lias  gene  into  studying  the  m.ilitary  man  as  an  inipcrtant 
resource  in  our  society  today,  t articular ly  with  the  td,anges  beim: 
affected  new,  but  tfiese  studios  would  indicate  tliat  one  of  thc!  nost 
..:u.:-aJ  varialiles  in  a^-t.i  i ni  :.q  full  value  of  the  individual  to 
Hocietv  has  liecn  at  locist  cua li ta t i vel y ignored. 

io  u.uderr,ta;'d  tlie  role  a military  wife  plays  in  her  liusJiaid'n 
life,  o.".o  m.o.st  first  analyse  her  role  in  thc  military  and  then 
exp'and  this  to  examine  how  her  position  as  wife  and  her  attitudes 
as  a member  of  the  military  (wives  in  respiondinq  to  (juestions  tend 
to  ansv;ct  "v/hen  ^ were  stationed  at  McConnell"  cr  "w  have  beer,  in 
the  service  7 years")  relate  and  exert  influence  within  thc  family. 
This  study  was  undertaken  for  just  sucti  purpose.. 


Hub : 

i'ao  •=>;  t’re  sample  was  drawn  from  the  wives  of  m.en  assicned  to  the 
381st  .'ttrateqic  .Missile  Vlir.q,  stationed  at  McConnell  Air  Force  Base. 

The  vhng  consisted  of  two  operations  squadrons,  one  missile  maintenance 
squadron,  a communications  squadron,  a security  police  squadron,  and 
various  sufport  squadrons  and  groups.  Tliere  were  approximately  IbOO 
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men  in  the  wing  during  the  time  of  the  study,  over  half  of  which 
were  married. 

Data  collection 

A series  of  three  questionnaires  were  administered  to  the 
subjects. 

The  first  questionnaire  (SDS-1)  was  developed  as  an  exploratory 
instrument  to  investigate  the  basic  premise  that  wives  of  Air  Force 
personnel  do,  in  fact,  have  strong  opinions  about  their  husband's 
military  affliation.  SDF-1  contained  questions  from  major  catagories: 
Demography;  Relations  with  Facilities,  Eenefits,  and  Services;  and 
General  Attitudes  toward  Militarv  Life.  An  open  ended  section  for 
general  comm.ents  was  also  included. 

As  a pilot  study,  SDS-1  was  distributed  to  tiie  Hospital  Squadron 
to  check  on  its  applicability  to  the  Air  Force  life  style.  L’pon 
analysis,  the  instrument  was  determined  to  be  applicable. 

The  first  questionnaire  was  administered  by  distributing  the  form.s 
to  all  the  married  men  in  the  wing,  and  requesting  the  husbands  to  take 
the  instruments  to  their  wives.  Return  envelopes  were  provided  to 
facilitate  the  return  of  the  completed  forms.  A total  of  827  copies 
of  SDS-1  were  distributed  on  April  14,  15,  16,  1971.  Of  these,  264 
completed  forms  were  returned,  creating  a response  rate  of  31.9%. 

Wives  Attitude  Survey  II  (WAS  II)  was  generated  for  two  major 
purposes.  The  first  section  was  aimed  at  investigating  in  depth  trends 
that  were  evident  in  the  open  ended  response  section  of  the  first 
survey.  The  second  aim  was  to  measure  more  precisely  the  relations  the 
wives  had  with  the  facilities,  benefits,  and  services.  Again,  the 
instrument  included  a demographic  section  and  an  open  response  section. 


377 


A listing  of  all  home  addresses  were  obtained  throuoh  the 
Consolidated  Base  Personnel  Office.  The  second  survey  was  mailed 
directly  to  the  wives  cn  .‘•'arch  13,  1972.  Included  in  the  m^-iling 
were  stamped,  self  addressed  return  envelopes. 

Wive's  .Attitude  Purvey  III  was  created  ruinarily  to  explore 
in  more  depth  the  relation  between  the  wife  and  the  facilitie.s, 
services,  and  benefits  provided  by  the  Air  Force.  This  i.nvestioatici. 
includi'd  further  inquiry  into  the  source  of  the  v.’i'/cs'  knowledge  and 
the  wives'  perceived  need  for  the  Air  Force  to  offer  tliose  i'-oms. 
Demographic  information  and  general  comment  sections  Wf'rc  included. 
Tlie  questionnaire  was  di.stributed  on  July  lb,  1972  in  the  s.am»-  manner 
as  V.'AP  II. 


Statistical  .Methods  Lmployed 

Due  to  the  disparate  nature  of  the  data,  tv/o  separate  techniques 
for  analysis  were  used.  A factor  analysis  was  performed  on  variables 
1-25  and  50-55  of  WAS  II  to  determine  the  dynamics  at  work  in  tlie 
attitudes  of  t'le  wi’'cs.  A discrini  rate  function  analysis  was  porformec 
in  A stepwise  m.anner  across  pairs  of  ten  selected  groups  and  subgroups 
of  the  sample.  The.se  groups  were  operational  defined  by  reflection  of 
their  husband.s  service  in  the  Air  Force  as  follows: 

Missile  Crew  - rf'n  that  arc  assigned  to  operational  Combat  Missile  Crows. 

Non  Missile  Crew  - men  that  are  not  assigned  to  Combat  Missile  Crews. 

Career  - men  vuth  five  years  or  more  of  time  in  service. 

Officer  - m.en  of  rank  0-1  or  above. 

Knl i.stcd  - men  of  rank  E-9  or  below. 

First  Term  Cfficer  - men  with  four  years  or  less  of  time  in  service 

with  rank  0-1  or  above. 
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Career  Officer  - men  with  five  years  or  more  of  time  in  service  with 
rank  of  0-1  or  above. 

First  Term  Enlisted  - men  with  four  years  or  less  of  time  in  service 

with  rank  E-9  or  below. 

Career  Enlisted  - men  with  five  years  or  more  of  time  in  service  with 
rank  E-9  or  below. 

The  dyads  selected  for  analysis  to  determine  intergroup  differences 
were:  Missile  Crew-Non  Missile  Crew;  First  Term-Career;  Officer-Enlisted; 

First  Term  Officer-Career  Officer;  First  Term  Enlisted-Career  Enlisted; 
First  Term  Officer-First  Term  Enlisted;  Career  Officer-Career  Enlisted. 

RESULTS 

Factor  Analysis  of  Wives'  Attitudes 

The  Factor  Analysis  performed  on  section  of  WAS  II  generated  ten 
factors.  The  contributing  variables  and  their  loadings  for  each  of  the 
factors  are  shown  in  Appendix  1.  The  first  factor  was  described  as 
passive  alientation/integration.  It  canoted  a passive  role  for  the  wives 
of  Air  Force  personnel.  They  apparently  did  not  feel  that  their 
participation  was  required  or  even  solicited,  yet  neither  did  they  feel 
that  they  were  rejected  or  prevented  from  becoming  involved.  This 
dynamic  displayed  the  disavowal  of  personal  responsibility  for  integration 
into  the  Air  Force  lifestyle  and  an  attitude  of  simply  "floating  along 
with  the  current." 

Factor  II  was  identified  as  a desire  for  information  versus  apathy. 
This  continum  was  between  a desire  for  more  information  about  how  the 
Air  Force  affects  her  life  and  the  apathy  which  is  present  in  all  walks 
of  life.  These  feelings  were  active  in  that  the  dynamic  stretches  from 
apathy  to  curiosity.  It  also  appeared  that  there  was  a recognizable 
solidarity  of  interest  in  what  the  Air  Force  was  and  did. 
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The  dynamic  at  work  in  Factor  III  could  most  aptly  have  been 
labeled  familial  maturity  and  independence  fvom  the  Air  Force/ 
familial  immaturity  and  dependence  on  the  Air  Force.  The  dynamic 
was  one  of  arowth  and  change  in  focus  of  attention.  As  the  family 
matured,  the  wife  became  more  interested  in  its  development  and  less 
interested  in  her  relationship  to  the  Air  Force. 

Factor  IV  providea  an  insight  into  the  v/ives'  perceptior.  of 
the  current  societal  trend  of  distaste  for  the  military.  The  continur 
traversed  the  area  between  prideful  identification  with  her  spouses' 
job  and  the  couple's  relationship  to  the  Air  Force  to  apologetic 
recognition  and  rejection  of  identification  with  the  Air  Force. 

Tl'iC  description  of  Factor  V was  a facet  of  wives'  attitude 
that  was  not  directly  related  to  the  Air  Force  as  a functioning 
orgar.ication.  Instead,  it  was  an  indication  of  the  group  identificati 
disassociation  among  the  wives  if  Air  Force  personnel.  This  within 
group  bipolarity  was  apparently  a very  important  reality  cf  life  in 
the  Air  Force  community. 

The  dynamic  exposed  in  Factor  VI  was  difficult  to  interpret.  With 
out  referring  to  causality,  it  appeared  that  physical/psycholocj ical 
separation  or  proximity/identification  attitudes  were  present  in  the 
sample . 

Factor  VII  probably  revealed  an  experential  attitude  set  about 
regulaticn/restriction  by  the  Air  Force.  The  dynamic  between  wives' 
feelings  of  personal  freedom  of  action  and  restriction  of  action  was 
quite  obvious. 
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The  eighth  factor  was  interpreted  as  two  varying  perceptions  of 
the  adage  "rank  hath  its  priviledges  (power)".  It  was  based  on  the 


perceived  transference  of  the  husband's  rank  to  the  wife.  The 
dynamic  was  primarily  one  of  perceived  power  in  the  transference 
as  opposed  to  no  power.  The  younger,  lower  rank,  wives  felt  more  power 
was  inherent  in  the  transfer,  while  the  older,  higher  ranking  wives 

1 

were  frustrated  by  t)ie  erosion  of  their  preconception  with  the 
realization  of  the  lack  of  power  that  the  husbands'  rank  gives  to 
his  wife. 

Identification  with  the  source  of  information  about  the  Air 
Force  was  the  interpretation  of  Factor  IX.  It  became  apparent  that 
as  the  v/ife  learned  more  about  the  system  with  which  she  is  involved, 
she  identified  more  closely  with  the  source.  The  poles  of  this 
continum  were  the  Air  Force  itself  and  her  spouse. 

The  last  factor  was  viewed  as  the  conscious  commitment  of  the 
I wife  toward  involvement  or  noninvolvement.  The  frustration  dimension 

1 did  not  enter  into  this  dynamic,  as  participation  was  directly  related 

[ to  the  personal  decision  of  the  wife. 

i 

I 

I Discriminant  Function  Analysis  of  Selected  Groups  and  Subgroups 

(Note:  The  nature  of  the  data  array  required  that  separate 

discriminant  function  analysis  be  performed  on  each  of  the 
three  instruments.  T);e  voluminous  nature  of  the  resulting 
analysis  (21  pages  of  tables)  precludes  any  but  a cursory 
treatment  of  the  findings  here.  The  interested  reader 
is  referred  to  the  tech  report  «)03  "The  Air  Force  Wife  - 
A Study  of  Morale  among  Military  Dependents"  issued  by  the 
Center  for  Human  Appraisal,  Wichita  State  University  for  a 
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detailed  ar.d  conprehensive  analysis  of  this  data.) 

'■'A ceil,'  Missile  Crete.  - The  group  of  wJves  whose 

husbands  were  in  Combat  t’.issile  Crews  were  discriminably  different  j 

than  the  rest  of  the  test  population.  The  main  factor  underlying  I 

this  difference  appeared  to  be  a lack  of  association  betv/een  the 
wife  and  the  Air  Force,  emphasized  by  a negative  attitude  on  the 
part  of  the  Fissile  Crew  wives  toward  facilities,  benefits,  and 
services  offered  by  the  Air  force . 

iirr^  - The  many  obtained  differences  between  the 

First  Terr,  and  Career  groups  could  be  expected  because  of  the 
obvicus  age  ar.d  time  in  the  service  differentials.  The  first 
tendan.-'v  wa.s  for  the  First  Term  wives  to  be  more  immature  than 
their  oldei  counterparts,  with  the  difference  showing  up  in  varied 
yerceptiens  of  facilities  and  services  offered  to  them.  The  Career 
wife,  since  she  had  m.oro  time  in  the  service,  was  more  familiar 
with  the  benefits  that  were  available,  and  maintained  a more 
p^ositive  attitude  toward  them. 

Cj'fia.  r-Er.listc  1.  - in  comparison  to  the  enlisted  wives,  the 
wives  of  officers  were  ba.sically  more  socially  orineted  and  participated 
to  a greater  degree  in  the  Air  Force  centered  community.  In  conjunction 
with,  this  ^'anticipation,  the  Officer  Wife  was  more  cognizant  of  the 
borofits  a'-ailable  on-base,  and  was  much  more  selective  in  the  use  she 
made  of  Air  Force  offerings.  These  inclinations  result  in  the  observation 
that  the  Officer  wife  identified  more  with  tlie  Air  Force. 


382 


1 

I 

First  Terr-.  Offityer-Career  Officer.  - In  the  subgroup  comparison 
of  First  Term  Officer  wives  and  Career  Officer  wives  the  results  were  | 

that  the  career  wives  identified  more  closely  with  the  Air  Force, 
knew  more  about  it,  and  oenerally  maintained  a stronger  relationship 
with  the  Air  Force.  The  findings  that  the  career  wives  were  more 
knowledgeable  was  further  emphasized  by  the  fact  that  the  First  Term 
Officer  group  was  more  influencable  by  military  propaaanda. 

First  Terr  Er.li sted-Career  Erlisted.  - Though  the  First  Term 
Enlisted  and  Career  Enlisted  wives  appeared  to  differ  in  primarily 
the  same  ways  as  the  First  Term  and  Career  Officers,  the  distinction 
was  slightly  different.  The  First  Term  Enlisted  had  a more  negative 
opinion  of  the  Air  Force,  along  with  the  expected  pattcrn.s  of 
disassociaticn  and  ignorance  of  the  Air  Force. 

First  Teir;  Officer  - First  Terr  Enlisted.  - The  differentiation 
shown  between  the  First  Term  Officer  and  tlie  First  Term.  Enlisted 
drew  along  the  same  line  as  the  overall  rank  differential.  The 
Officers  wives  v;ere  more  associated  witi;  the  Air  Force,  though  there 
was  still  a significant  thread  of  disinterest  in  the  discrimination. 

Again,  greater  maturity  was  noticable  in  the  wives  of  the  First  Term 
Officers,  and  larger  self  percef:tion  of  independence. 

Career  Officer-Career  Erl '.’'•I ed.  - Again,  the  difference  between 
the  Career  Officer's  wife  and  the  Career  Enlisted  wife  appears  to  have 
been  one  of  rank.  The  Officer  wife  felt  more  socially  interested  in  the 
Air  Force,  and  was  generally  more  associated  with  the  Air  Force,  Further, 
the  Officer  wife  appeared  to  make  more  conscious  decisions  about  which 
facilites,  benefits,  and  services  were  beneficial  to  her,  and  did  not 
appear  positively  inclined  toward  something  merely  because  it  was 
'Air  Force . ' 


DISCUSSION 


Although  numerous  complaints  were  voiced,  the  wives  for  the 
most  part  had  a favorable  attitude  toward  the  military  lifestyle 
and  their  participation  in  it.  In  fact,  most  of  the  wives  expressed 
a desire  for  more  information  about  the  Air  Force.  The  study 
indicated  the  average  AF  wife  knows  very  little  about  the 
facilities,  benefits  and  services  available  to  her. 

Nevertheless,  the  dependent  briefings,  designed  to  offer  this 
type  of  information,  seemed  to  be  an  irritant  to  some  wives, 
especially  the  younger  group.  A fairly  common  complaint  about  the 
dependent  briefings  was  the  "impersonal"  manner  in  which  the  wives 
were  treated.  Many  also  disliked  the  term  "briefing"  The  term 
seemed  to  accentuate  the  military  atmosphere  of  the  meetings. 

Because  of  this  attitude,  many  of  the  wives  said  they  did  not  attend 
the  dependent  briefings  regularly,  which  forced  them  to  look  for  an 
alternate  inform.ation  source. 

This  alternate  source  usually  was  the  husband.  But  he  too  was 
often  an  insufficient  source  for  information.  For  one  thing, 
the  husband  doesn't  know  the  kinds  of  things  his  wife  wants  and  needs 
to  know  and  doesn't  appear  to  be  interested  enough  in  these  areas 
to  find  out. 

The  information  factor  should  be  of  major  importance  to  AF  officials 
as  the  survey  also  indicated  the  wives,  as  a group,  tend  to  identify 
and  form  attitudes  about  the  AF  based  on  the  information  they  received. 
However,  the  survey  also  indicated  that  coercion  would  NOT  be  a good 
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means  of  attaining  the  desired  goals.  That  is,  requiring  wives  to 
attend  dependent  briefings,  undoubtedly  would  have  a very  negative 
effect. 

As  might  be  expected  the  wives  of  men  with  long  term  associations 
with  the  AF  tended  to  have  a prideful  identification  with  the  Air  Force. 
Some  of  the  younger  wives,  however, seemed  apologetic  about  their 
husband's  military  affiliation.  This  may  be  due  to  the  recent  negative 
societal  view  of  the  military  among  some  young  people. 

The  survey  indicated  that  the  wives  who  identified  with  the  AF 
lifestyle  tended  to  live  close  to  the  base,  and  those  least  interested 
in  that  lifestyle  lived  further  from  the  base.  However,  it  is  not 
possible  from  this  study  to  determine  which  is  the  cause  or  which  is  the 
result  of  this  phenomenon. 

It  was  also  found  that  wives  with  no  children  tended  to  depend 
on  the  military  to  give  structure  to  their  lives.  However,  as  children 
came  into  the  family  unit,  the  wife  changed  her  viewpoint  and  became 
more  involved  with  her  family  and  less  involved  with  the  military. 

An  interesting  phenomenon  revealed  by  the  survey  was  the  concept 
some  of  the  wives  had  about  rank  transference.  It  appeared  that  wives 
of  lower  ranking  men  perceived  a great  deal  of  power  transferred  to  the 
wives  of  higher  ranking  men  (both  officer  and  enlisted.)  However,  when 
the  men  rose  into  the  higher  ranks,  the  wives  realized  there  was  very 
little  real  power  transferred  to  the  spouse.  The  frustration  seemed 
to  increase  as  the  husband's  rank  increased  and  the  realization  that 
she  had  no  real  power  became  more  evident. 
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The  various  group  comparisons  made  in  the  study  revealed  that 
the  feelings  and  attitudes  of  the  wives  tended  to  be  group 
specific . 

For  example,  the  group  of  wives  with  husbands  in  missile  crews 
had  the  most  negative  feelings  about  the  Air  Force  and  its  benefits 
and  facilities.  This  was  not  surprising,  however,  as  our  previous 
Career  Attitude  Survey  showed  missile  crew  members  to  be  among  the 
least  satisfied  men  in  the  AF . 

One  factor  that  may  have  contributed  to  the  poor  attitude  of  the 
"missileer's"  wife  was  the  fact  that  her  husband's  duty  required  him 
to  be  separated  from  her  overnight  several  nights  a week.  This 
factor  may  be  compounded  by  the  fact  that  most  missile  crew  members 
were  first-termers  and  were  relatively  young.  The  wife's  younger 
age  may  have  tended  to  make  her  less  understanding  about  her  husband's 
reoccuring  absence. 

Wives  of  the  career  group  men  were  more  familiar  with  the  bene- 
fits available  to  them  and  had  a ’•.lore  positive  attitude  toward  them 
than  the  first  term-wives. 

In  comparison  to  the  enlisted  wives,  the  wives  of  officers  were 
basically  more  socially  oriented  and  participated  to  a greater  degree 
in  the  AF  centered  community. 

The  career  officer  wives  identified  more  closely  with  the  AT,  knew 
more  about  it,  and  generally  had  a stronger  relationship  with  the  Air 
Force  than  the  first-term  officer  wives.  The  first  term  officer  group 
also  seemed  to  be  more  readily  influenced  by  military  propaganda. 
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The  difference  between  the  career  officer  wife  and  the  career 


enlisted  wife  appeared  to  be  one  of  rank.  The  career  officer  wife 
felt  more  socially  interested  in  the  AF  and  generally  was  more 
associated  with  the  AF.  Also,  the  officer  wife  did  not  appear  to 
approve  of  a program  or  benefit  merely  because  it  was  'Air  Force.  ' 
By  dividing  the  wives  into  four  groups  it  was  possible  to  rank 
them  according  to  their  feelings  about  the  AF . The  career  officer 
wife  was  the  most  favor. ibiy  inclined  towards  the  AF,  followed  by 
the  career  enlisted,  first  term  officer,  and  first  term  enlisted 
wife. 


IMPLICATIONS 

Overall,  it  would  appear  that,  because  of  her  obvious  effect 
on  the  husband,  the  AF  wife's  somewhat  positive  feelings  about  the 
Air  Force  and  her  desire  for  more  information  are  desirable  effects. 

Her  apparent  desire  to  become  more  of  a part  of  the  military 
centered  community  seems  to  be  blocked  because  of  a lack  of  an  easy 
aveneue  to  do  so.  The  results  seemed  to  indicate  that  both  she 
and  the  Air  Force  lacked  the  real  initiative  to  draw  her  into  the 
community. 

The  apparent  failure  of  the  dependent  briefings  in  providing 
information  to  the  wives  should  be  given  some  attention.  It  seems  that 
a more  "feministic"  approach  is  desired  by  the  wives. 

In  conclusion,  it  seems  evident  that  the  AF,  as  well  as  the  other 
services,  is  ignoring  a segment  of  its  community  that  is  an  extremely 
powerful  factor  in  retention  and  possibly  an  equally  powerful  influence 
on  job  performance  and  morale  eimong  the  men. 
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Factor  I:  Passive  alienation/integration 


Variable 

Number 

Variable 

Description 

Factor 

Loading 

4. 

The  A.F. 
think. 

doesn ' 

't  care  what 

the  wive's  of  its  personnel 

-0.889 

13. 

The  A.F. 

doesn ' 

't  care  what 

I think. 

-0.884 

7.  There  are  no  procedures  for  me  to  express  my  feeling  -0.788 
about  A.F.  policies. 

10.  Civilians  don't  respect  military  personnel.  -0.570 

19.  The  A.F.  keeps  the  wives  of  its  personnel  well  informed.  0.559 

24.  Life  as  an  A.F.  wife  provides  me  many  opportunities  to  0.402 
get  involved. 

5.  A.F.  wives  should  be  kept  better  informed  of  base  act-  -0.369 
ivities . 


Factor  II:  Curiosity/apathy 


Variable 

Number 

Variable  Description 

Factor 

Loading 

15.  If  I understood  it  more,  I think  the  A'r  Force  would  0.739 

be  interesting. 

14.  I would  like  to  be  invited  to  attend  my  husbands  re-  0.707 

enlistment  interviews. 

9.  I don't  want  to  know  more  about  my  husband's  job  than  -0.701 
I already  know. 

8.  Base  activities  have  a direct  affect  on  me.  0.526 

5.  A.F.  wives  should  be  kept  better  informed  of  base  0.374 

activities. 

11.  Most  wives  think  they  have  their  husband's  rank.  0.322 
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Factor  III;  Familial  maturity/immaturity 


Variable  Factor 

Number Variable  Description Loading 

51.  How  long  have  you  been  in  the  service  (4-8yrs.:5)  ' 0.860 

53.  How  many  children  do  you  have?  (1.2)  0.788 

52.  Do  you  expect  your  husband  to  make  the  A.F.  a career  -0.727 
(probably) 

22.  The  A.F.  should  not  be  a 24-hour  a day  job.  0.590 

24.  Life  as  an  A.F.  wife  provided  me  many  opportunities  -0.480 

to  get  involved. 

1.  Wives  of  A.F.  personnel  should  be  involved  in  formul-  0.360 
ating  A.F.  policy. 

11.  Most  wives  think  they  have  their  husband's  rank.  0.305 

*********************************************************************** 

Factor  IV:  Prideful/apologetic  identification 

Variable  Factor 

Number Variable  Description Loading 

20.  A wife  should  be  proud  of  her  husband's  profession.  -0.889 

25.  The  A.F.  should  not  be  just  another  job.  -0.553 

5.  A.F.  wives  should  be  kept  better  informed  of  base  -0.413 

activities . 

11.  Most  wives  think  they  have  their  husband's  rank.  -0.340 

*********************************************************************** 
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Factor  V:  Within  subculture  identif ication/disassociation 


Variable 

Number 


Variable  Description 


18.  A.F.  wives  have  a lot  in  common. 

12.  A.F.  wives  have  a number  of  similiar  problems. 

17.  I often  feel  I am  a member  of  the  A.F. 

2.  I enjoy  associating  with  other  A.F.  wives. 

21.  I know  the  wives  of  members  of  my  husband's  unit 
fairly  well. 

24.  Life  as  an  A.F.  wive  provides  my  many  opportunities 
to  get  involved. 


Factor 

Loading 

0.825 

0.722 

0.702 

0.565 

0.540 


0.474 


Factor  VI:  Physical  & psychological  proximity/separation 


Variable 

Number 


Variable  Description 


Factor 

Loading 


54.  How  far  do  you  live  from  base?  (3-5mi) 

1.  Wives  of  Air  Force  personnel  should  be  included  in 
formulating  Air  Force  policy. 


0.885 


-0.370 


Factor  VII:  External  restriction/freedom 


Variable 

Number 


Variable  Description 


Factor 

Loading 


6.  Wives  of  Air  Force  personnel  are  free  to  do  what 
they  want. 


0.790 


3.  Many  of  my  personal  activities  are  regulated  by  Air  -0.498 

Force  policies. 

************************************************************************ 
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Factor  VIII:  Rank  transference  aspiration/frustration 


Variable 

Number 

Variable  Description 

Factor 

Loading 

55. 

What 

is  your  husband's  rank. 

0.832 

24. 

Life 

ities 

as  an  Air  Force  wife  provides  me  many 
; to  get  involved. 

opportun- 

-0.396 

11. 

Most 

wives  think  they  have  their  husband's 

rank. 

0.367 

Factor  IX:  Identification  with  information  source 


Variable 

Number 

Variable  Description 

Factor 

Loading 

26.  Where  did  you  learn  the  most  about  the  Air  Force.  0.880 

1.  Wives  of  Air  Force  personnel  should  be  involved  in  0.270 

formulating  A.F.  policy. 


14.  I would  like  to  be  invited  to  attend  my  husband's  0.268 

reenlistment  interview. 

17.  I often  feel  I am  a member  of  the  Air  Force.  -0.250 
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Factor  X:  Personal  latitude  for  involvement/non-involvement 


Variable  Factor 

Number Variable  Description Loading 

23.  Air  Force  benefits  do  not  interest  me  a great  deal.  0.684 

16.  I don't  care  about  Air  Force  policy  except  as  it  0.581 

affects  me. 

5.  Air  Force  should  be  kept  better  informed  of  base  -0.442 

activities. 

1.  Wives  of  Air  Force  personnel  should  be  involved  in  -0.392 

formulating  Air  Force  policy. 
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ABSTRACT 

This  study  concerns  prediction  of  retention  in  training  at  an 
institution  where  voluntary  departures  are  more  frequent  than  academic 
failures,  and  the  prediction  of  voluntary  attrition  has  proved  to  be 
extremely  difficult. 

Usin^  the  cadet  intakes  of  1970  and  1971,  data  from  two  personality 
tests,  leader  performance  scores  from  cadet  wine  peer  evaluation  reports, 
and  biographical  information  from  a 92-item  Personal  Preference  and 
Biographical  Questionnaire  (PPBQ),  several  analyses  were  carried  out  to 
ascertain  those  variables  which  are  of  greatest  value  in  the  early 
identification  of  voluntary  leavers. 

Predictions  for  tne  first  year  of  training  of  the  intake  of  1^72 
were  made  based  on  information  gathered  in  the  previous  two  years,  and 
the  outcomes  are  reported  and  discussed. 
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CADET  iiLTlCNTIOK  AT  THE  KOYAL  /ITLTTARY  COLLEGE  OK  CANADA  AS  A KUNCTION 
OF  PERSONALITY,  LEADERSHTp  pF,RFOR»1ANCE,  A^^^.  !3IOGR;'vPHICAL  VARIABLES 

INTRODUCTION 


My  aim  today  is  to  tell  you  about  the  Royal  Military  College 
(RMC)  of  Canada  at  Kingston,  Ontario,  and  to  describe  a research  project 
which  studied  cadet  attrition.  We  have  found  that  academic  failure  is 
relatively  easy  to  predict  and  that  selection  standards  can  be  adjusted 
acccrdint^ly.  However,  at  the  moment,  the  rate  of  voluntary  withdrawal 
from  the  college  is  about  four  times  that  of  academic  failure.  The 
present  study,  tnerefore,  ir  concerned  with  voluntary  leavers. 

BACKGROUND  INFORMATION 


HMC  was  opened  nearly  100  years  ago,  was  patterned  after  W'est 
Point,  and  offers  U-year  university  prcffra-mmes  in  Engineering,  Science, 
and  Arts.  The  college  also  has  a graduate  school  of  Engineering  and  of 
V/ar  .Studies.  Hie  cadet  population  is  approximately  600.  The  programme 
is  both  physically  and  mentally'  demanding. 

METHOD 


The  followinf:  instruments  were  used  to  gather  data  with  which  to 
lest  tne  hypothesis  tnat  cadets  wno  withdraw  from  the  college  are  not 
different  :rom  thos<’  who  stay.  (Table  1 provides  information  on  time  of 
testing  and  retcs+mg  if  applicable.) 

1.  Cattell's  16PF. 

?.  Jackson's  Personality  Research  Form. 

3.  RMC  Personal  Preference  and  Biographical  Questionnaire. 

U.  RMC  Cadet  Peer  Rating  Assessment. 

Sc^'eral  kinds  of  information  were  being  sought.  The  four  cat- 
egories listed  below  will  be  reported  separately  in  the  results; 

First,  I wanted  a comparison  between  university  students  in 
general  and  RMC  students  on  a well  known  personality  test  (l^PF). 

Second,  I wanted  to  see  if  at  the  time  of  recruiting  the  even- 
tual leavers  could  be  discriminated  from  stayers  on  the  basis  of 
personality,  or  cio^raohical  information.  (lAPF,  PRF,  and  PPBQ) 

Thi”d,  I wanted  to  examine  the  value  of  leader  performance  peer 
ratings  in  identifying  leavers.  (PEER) 

Fourth,  T hypothesized  that  all  cadets  would  change  as  a 
function  of  both  maturation  and  the  military  socialization  process;  I 
also  wanted  to s ee  if  leavers  changed  in  a different  way  tnan  stayers. 
(l^PF,  PEER) 
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Personality  inventories  were  analyzed  using  t-test  and  analysis 
of  variance  procedures,  while  Chi-square  tests  of  significance  were 
used  in  the  examination  of  the  Personal  Preference  and  Biographical 
Questionnaire  responses.  These  were  also  analyzed  by  cluster-analytic 
techniques,  some  of  which  were  developed  at  RMC,  and  by  traditional 
methods  of  item  analysis.  Data  analyses  were  carried  out  by  members  of 
the  department.  Full  reports  are  referenced  under  Bain  (1972,  1973), 
Shields  (1972,  1973) > and  Carpenter  (1972,  1973). 

RESULTS 


I snail  report,  the  results  by  study  and  for  each  instrument 
employed  in  tnat  particular  study. 

Study  ^1.  Comparison  oi  'iC  students  with  Cattell's  norms. 

Table  2 displays  the  means  and  standard  deviations  of  the  RMC  sample 
and  Cattell's  normative  data.  In  comparison  with  other  university 
students,  RMC  students  are  more  reserved,  intelligent,  stable,  con- 
scientious, tough  minded,  trusting,  imaginative,  forthright,  self- 
assured,  conservative,  self-sufficient,  and  controlled. 

Ctudy  ^2.  Selection  ua+a  relationship  with  stay/leave  criterion. 
2 . 16PF . Table  3 shows  tne  means  for  stayers  and  leavers  at  time  of 
recruitini^.  There  are  unree  significant  differences.  Stayers  are  more 
self-assured,  oontrcllea,  ana  relaxed  than  leavers. 

2.  PRF.  Table  u shows  the  means  of  stayers  and  leavers  at  time  of 
recruiting.  Tr.ere  are  no  significant  differences. 

3.  PPBQ,  Analysis  of  tne  PPBQ  has  provided  the  most  interesting 
results  to  date.  Using  octn  cluster  analytic  and  correlational  tech- 
niques, three  scaler  (voluntary,  military,  and  academic)  were  produced 
and  both  methods  produced  multiple  correlations  of  approximately  .60 
with  the  stay/leave  criterion,  and  a correlation  of  ,li7  between  the 
voluntary  scale  alone  and  the  J/L  criterion.  The  stability  of  this 
result  is  being  tested  by  cross-validation  and  the  results  so  far  are 
shown  in  Table  P. 

Study  v3.  Liffercnces  between  stayers  and  leavers  on  PEER 
Ratings.  Table  6 shows  tae  ,”ean  scores  on  the  15  cricical  requirements 
o*'  leadership  for  each  of  the  two  groups.  Stayers  exceed  leavers  on 
all  1^,  and  2.1  of  tiese  reacn  significance. 

Although  the  PEER,  rating  is  of  value  in  identifying  potential 
voluntary  withdrawals,  it  cannot  be  used  as  a screening  device  at  the 
Recruiting  Centre  because  a period  of  about  two  weeks'  intimate 
acquaintance  is  requlreu  tefore  peers  have  any  substantial  information 
on  wh:ch  to  make  Judgements. 
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study  #!(.  Change  as  a function  of  maturation  and  socialization. 
Table  7 shows  the  results  of  three  administrations  of  the  l6PF a t one 
year  intervals.  All  the  changes  that  are  going  to  take  place  (with  one 
exception)  r.appen  in  the  first  year.  Cadets  become  more  "affected  by 
feelings",  "happy-go-lucky",  "expedient",  "tender-minded",  "suspicious" 
"imaginative",  "apprehensive",  "undisciplined",  and  "tense".  The  one 
exception  is  "forthright",  wnere  on  the  third  testing  the  group  bounced 
back  toward  the  "shrewd"  end  of  the  scale. 

Differential  cnanges  between  stayers  and  leavers  are  as  follows; 

1.  16PF.  Table  8 provides  tnis  information  in  the  Groups  X Trials 
interaction.  The  two  groups  differ  differentially  on  the  forthright- 
shrewd  factor.  Leavers  become  more  forthright  while  stayers  become 
more  shrewd. 

2,  Peer  Rating.  Table  9 shows  that  only  one  Leadership  critical 
requirement  is  significant  on  the  G X T interaction,  ie,  "interest  and 
motivation".  On  it,  leavers  decline  while  stayers  rise.  These  data 
were  replicated  using  the  November  and  April  peer  ratings. 

DISCUSSION 

In  the  field  of  testing,  a significant  relationsr.ip  is  not 
necessarily  a useable  relati onshin.  None  of  the  personality  tests 
provided  anythinr  that  a serious  researcher  would  recommend  to  his 
patron  as  a selection  device.  The  peer  rating  assessment  is  assuredly 
of  values  in  identifying  cadets  who  are  having  problems,  but  must  be 
used  as  a counselling  rather  tnan  a selection  instrument  because  of 
the  administration  lag  time. 

The  PP3Q  alone  appeared  t o have  promise  as  a selection  in- 
strument, The  original  correlation  between  test  and  stay/leave  shrank 
from  .u7  tc  .20  in  cross-validation. 

CONCLUSION 


Person.ulity  factors  nave  an  elusive  relationship  with  real-world 
measurable  criteria.  Despite  the  lack  of  useful  relationships  in  tnis 
study  we  cannot  abandon  tnat  strong  intuitive  belief  that  a great  deal 
01'  tne  variance  in  success  and  failure  can  be  accounted  for  by  personal 
ity  variables,  and  that  given  a more  appropriate  research  design  and 
an  adequate  criterion  the  importance  of  personality  factors  could  be 
demonstrated.  The  discovery  that  the  PP3Q,  a "nome-made"  test  was  the 
oest  in  the  oattery  is  a heartening  experience,  and  a recommendation 
for  test  constructors,  to  p«»rsevere.  Further,  ’/rtien  it  is  considerec 
that  candiaates  for  R’-C  are  a nighly  selected  sample,  relationships 
witnin  tnat  sample  are  sure  to  suffer  from  the  effects  of  restriction 
of  range. 
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Finally,  the  development  of  novel  methodological  approaches  to 
the  problem  of  using  non-linear  (categorical)  data  to  predict  categorical 
criteria  is  potentially  of  great  value  in  a wide  variety  of  prediction 
situations. 


I 
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TABLE  ] 


TESTS  AND  TTf'E  GE  ADMINISTRATION 


HvSTRUMBv’T 

'irTIAL  TEST 

RETEST 

Cattell'.T  1 '’■'PF 

i'nit  or 

Ist  wppk  at  college 

On  release  or  as  control 
subject,  and  annually. 

.JacKi  n'r  PF-i' 

Recr  .■’-1  in’  unit  or 
let  week  at  college 

On  release  or  as  control 
subject. 

PFGQ 

Is*  vec"  at  college 

rl'C  GfRpt  ?ef:r 
Ratiru^ 

At’t»r  2 weeks 
a:.  cclle.-;e  (Aug) 
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TABLE  5 


RF:3'JLTS  of  FPBQ  analysis^! 

1.  Va]  idation  Oainple  1970  + 1971  N * 229 

V A M 
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' .372  1 

.173 

.165 
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-.016 

correlation  of 

.59ii  to  : 

^ 1972 

N - 127 

( using  V 

r = .20 


Note:  V - vol  inUiry  attrition  scale 

A = academic  attrition  scale 
M = military  attrition  scale 
S/L  = stay/leavp  criterion 
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MOTivAiio:;  OF  EF.i.isxr.D  fj:3i;rvi3T3  i::  a koe'-deatt  Ar.iosp:it:rj: 


BY 

LT  cyzfbyI 

catle::r  rATTi'Y;:.  a:,:.'  * r:;ccn.u's  br.'_;c:: 

divisio:: 

U.  S.  COAST  GUAJID  HE. AD  QUARTERS 
:’A5Hi.;r.To;:,  d.  c. 


As  people  involved  in  the  training  and  evaluation  of  military  personnel, 
we  are  today  •..'crning  in  a iiostile  atmosphere  - a time  wnen  our  services 
face  a manpower  crisis  both  in  quality  and  numbers.  Hallowed  traditions 
are  falling  hy  the  wayside  at  military  academies,  enlisted  people  are 
asking  '''.d-.y?"  more  often  and  prcspective  military  personnel  remember  Viet- 
nam with  some  distaste.  The  draft  has  disappeared  as  an  inducement  for 
satisfactory  milita'^y  performance. 

As  the  number  of  personnel  on  active  duty  dv/indles,  emphasis  on  the  impor- 
tance of  a strong  reserve  force  grows.  Examples  reflect  situations  in  the 
writer's  service.  Hoxi;ever,  it  is  hoped  that  the  concepts  illustrated  have 
more  general  applicability. 

Definitions  and  Assumptions: 


1.  A reservist  who  is  well  trained  in  his  rating  (field  of  specialization) 
is  well  trained  for  mobilization. 

2.  A.ugir.entation  consists  of  any  reserve  activity  that  supports  effective 
training  for  mobilization  'while  meeting  a stated  need  of  an  element  of  thiC 
regular  Coast  Guard. 

3.  Goal; 

a.  Provide  (officers  and)  petty  officers  qualified  to  perform  ".X" 
effectively. 

b.  Retain  qualified  reservists. 

4.  "X"  is  defined  as  a ccmposjmic  cf  technical,  practical  and  cormon  sense 
traits  PLUS  leadership,  loyalty,  and  sense  of  responsibility  required  to 
perform  satisfactorily  in  a billet  that  meets  a stated  need  of  the  regular 
component. 

A great  deal  of  work  has  been  done  in  identifying  the  technical  and  practical 
traits  monticnod  in  t!ie  definition  of  "X".  Leadership  schools,  patriotic 
pubiic  itieu'^,  coercion  and  other  '.'ays  have  ocen  v.'idely  used  to  instill  other 
desirahi.^  queliti';s.  Let  as  aas'.m.c  for  no'.;  that  th.e  required  teei’.nicai  rr.d 
practic.'il  ‘H  ills  Imve  been  iciontified.  As  trainers  and  evaluators  '.;e  desire 
to  (1)  provide  (LC'^LU' ;ic;  LLT)  enlisted  reservists  with  the  necessary  learning 
and  (2)  we  "ish  to  ’■iiA  i.vols. 

L'e  dcsi.re  lo  accomp  1 isa  uae  trainjeig  tenr.  ■.'ithout  resortir  , to  Uv.  :',.iv.' 
contives  such  an  threatening  cal!  to  ertonded  active  duty  as  a punitive  moarjure, 

^The  opinions  or  assertions  contained  herein  are  the  private  ones  of  the  writer 
and  are  not  to  be  constnied  as  official  or  reflecting  the  views  of  the  Commandant 
or  the  Coast  Guard  at  large.  411 


In  the  Coast  Cuart!  Reserve,  wc’  have  approximately  9500  enlisted  reservists 
in  37  ntin2'5.  T'.u  ••  arc  ucll  ceveated.  Over  20"  of  the  group  have  sixteen 
or  r.ji-e  years  cl  educatler. . 'v.r  ono-‘,’.alf  have  more  than  twelve  years  of 
education  and  over  2Ch  h.ava  hi_h.  acl'.ool  dipienas.  These  people  enlisted  for 
various  reasons.  Inti''  verv  recently  the  draft  iias  been  an  important  moti- 
vating facLor  in  both  enlistment  and  continued  satisfactory  participation. 

This  point  is  very  significant  far  reasons  which  will  become  apparent  in  a 
few  moments. 

Enlisted  advancements  are  earned  through  completing  a variety  of  requirements 
including  time-in-grade,  correspondence  courses  and  successful  competition 
in  servicewide  examinations.  These  exaras  are  the  same  as  the  ones  given 
Regular  personnel  and  are  an  important  part  of  maintaining  a "one  service" 
standard. 

Correspondence  course  completion  is  a prerequesite  to  ^3ervicewide  competition. 

Emphasis  is  placed  on  servicewide  examinations  here  because  it  gives  us  a 
relatively  objective  measure  of  training  needs  by  topic  area.  Further,  it 
is  felt  that  level  of  participation  is  a rough  indicator  of  interest  in  the 
program. 

Participation  in  correspondence  courses  and  servieex^ide  examinations  has 
fallen  sharply  in  the  last  wo  years.  P.easons  for  the  decline  include 
the  near  demise  of  the  Coast  Guard  Reserve  in  1971,  xvhich  caused  morale 
problems,  and  the  trend  av;ay  from  traditional  monthly  drills  in  a class- 
room atmosphere  - the  advent  of  augmentation. 

Enlisted  personnel  in  the  Coast  Guard  Reserve  are  distributed  through  the 
pay  grades  as  follov/s: 


TABLE  I 


Distribution  of  Enlisted  Personnel  in  the  Coast  Guard 
Percentage  by  Pay  Grade,  30  July  1973, 


Rate 


Percentage 


E-4  and  Belox/  53.32 

E-5  21.87 

E-6  14.97 

E-7  and  Jiiove  9.84 


100.00 


Reserve  (Selected) , 


Desired  Percentage 

40 

30 

20 

10 

100 


Tais  siL.’.“om,  170'^  rues tionnai rc,-.  x.-ere  distributed  to  enlisted  rosorxxists, 
538  were  returned  and  the  proportion  returned  were  roughly  equivalent  by 
pay  grade  to  on-board  strength.  Table  Tx;o  indicates  response  to  pertinent 
questions  by  pay  grade. 
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TABLE  II 


Note  several  interesting  things: 

1.  The  higher  the  pay  grade,  the  greater  the  faith  in  the  advancement  system 
(reflected  in  the  question  "Do  you  think  the  servicewide  exam  system  is  fair?"). 

2.  The  relatively  low  participation  in  servicewide  exams  in  April  1973. 

3.  Indicated  applicability  of  using  civilian  vs.  military  related  skills. 

4.  Reenlistment  intentions  by  pay  grade. 

Now,  recall  the  personnel  distribution  by  pay  grade  and  compare  with  the 
reenlistraent  intentions  represented  here.  Note  that  the  LOLTST  expected 
retention  rate  is  in  the  most  highly  populated  pay  grades  which,  in  turn 
serve  as  a source  for  pay  grades  E-5  and  E-6  where  we  have  comparative 
deficits . 

Enlistment  dates  indicate  that  many  of  the  personnel  in  the  E-4  and  below 
category  enlisted  while  the  draft  was  still  active. 

The  obvious  challenge  is  retention  in  the  lower  pay  grades  and  the  encourage- 
ment of  advancements.  The  special  target  is  the  group  of  reservists  who  an- 
wered  "Jlaybe"  to  the  question  "Doy  you  plan  to  reenlist?". 

Now,  let  us  look  at  a pyramid  presentation  of  Maslow’s  hierarchy  of  needs. 
Proposal  1.  It  is  proposed  that  the  typical  reservist's  position  in  the 
hierarchy  has  shifted  upward  (or,  in  the  case  of  obligors,  will  shift  up- 
ward upon  completion  of  required  service)  in  the  absence  of  the  draft  and 
that  this  factor  must  be  addressed  if  a satisfactory  climate  for  personal 
performance  and  retention  is  going  to  exist. 

TABLE  III 


Dllemm.a;  This  approach  must  involve  a commitment  on  the  part  of  the  program's 
administrators  as  well  as  a change  in  written  directives,  guidelines  and  other 
communications.  Proposal  2.  Service  training  requirements  can  be  met  in  a 
way  that  also  contributes  to  the  reservist's  personal  needs.  The  concept  of 
working  toward  BOTH  goals  should  support  quality  participation  and  retention. 

Recall  the  definition  of  augmentation.  - "Augm.entatlon  consists  of  any  reserve 
activity  that  supports  effective  training  for  mobilization  while  meeting  a 
stated  need  of  the  regular  Coast  Guard."  A recent  survey  of  enlisted 
reservists  in  the  11th  Coast  Guard  District  measured  reenlistment  prospects 
for  obligated  and  non-obligated  reservists  by  type  of  augmentation  duty. 

The  varieties  of  augmentation  that  equated  with  highest  retention  potential 
among  obligors,  especially,  vrere  those  that  involved  active  identifiable 
accomplishment  - boating  saicty  and  search  and  rescue. 

TABLE  IV 

Less  subst  mtial  activitie..  did  nat  enjoy  the  same  implied  enthusiasm.  A 
fringe  benefit  of  augnentatien  is.  th.at  productive  work  is  being  acce  pi!  ...ev.  - 
not  typically  the  case  in  the  past  when  reservists  spent  drill  after  drill 
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in  clabtirooms  sleeping  through  irh  II  training  tilms. 


Admittedly,  .u.gr..er.taLloa  .ilone  is  not  lihely  to  produce  the  variety  of 
skills  recjuired  to  attain  "X"  as  defined  earlier.  Other  types  of  training 
must  be  incluuca  in  the  blend  to  attain  the  desirea  result.  This  mix  must 
contribute  to  the  reservist's  needs  as  well  as  military  requirements  if 
we  are  to  attain  our  goals. 

Let  us  now  look  at  some  specific  things  that  can  be  done  to  accomplish  the 
ends  set  forth: 

1.  Install,  use  and  maintain  an  effective  automated  personnel  accounting 
system.  This  management  tool  offers: 

a.  Ready  availability  of  data  to  personnel  action  staffs. 

b.  A convenient  index  of  available  reservist  skills. 

c.  Elimination  of  redundant  personnel  procedures  such  as  repeated  requests 
for  the  same  information. 

d.  Ready  identification  of  refresher  training  needs. 

e.  Predictor  information  for  advancement  eligibility,  retirements,  etc. 

2.  Take  advantage  of  getting  reservists  off  to  a positive  start  by  insuring 
quality  programs  and  well  trained  instructor  personnel  at  initial  training 
centers. 

3.  Personalize  the  reservist's  relationship  with  his  service  through  expanded 
use  of  direct  mailing,  prompt  attention  to  correspondence  and  more  direct 
contact  with  officer/advisors.  Does  the  reservist  have  civilian  skills 
compatible  with  service  needs?  Let's  use  them  unless  the  person  involved 
really  wants  variety.  (Assumption  - adequate  information  is  available  to 

the  reservist  for  him  to  make  an  intelligent  decision).  We  can  also  provide 
for  changes  in  field  specialization  where  justified.  Here  is  a real  oppor- 
tunity to  employ  a great,  relatively  untapped  resource,  the  reserv'e  junior 
officer . 

4.  PeriodicaJ ly  review  all  training  reqiurements  t.  insure  that  they  are 
relevant  and  realistic.  One  way  this  may  be  accomplished  is  by  taking  the 
following  approach: 


TABLE  V 


Evaluation  (standards,  requirements, 
measures  of  success)  retention,  servciewide 
analysis,  personal  needs 


Use  (needs  satisfaction) 

1.  Shoulti  support  advancement 
ard  T'  i.'ii. 
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For  example,  servicewide  exam  analysis  by  section  indicated  a need  to 
strengthen  skills  in  "military  factors"  or  knowledge  in  areas  applicable 
to  all  pnllsttJ  -personnel  in  all  ratings.  Ililitary  factors  cover  such 
topics  as  rc:U,  first  aid,  customs  and  courtesies,  etc.  and  are  traditionally 
regarded  as  "dickey  iouse".  Knowledge  of  military  factors  enhances  a 
reservists  change  for  earning  a qualifying  score  on  the  servicewide  exam. 

A way  to  learn  the  required  items  is  being  made  available  through  use  of 
a highly  portable  audio-visual  training  device  together  with  a quiz. 

5.  Maximize  use  of  "where  the  action  is"  training  resources  and  improve 
effectiveness  of  other  resources. 

6.  Offer,  so  far  as  is  possible,  an  equitable  opportunity  for  advancement 
and  opportunity  to  earn  retirement  benefits. 


Enlisted  reservists  are  an  intelligent  group  of  individuals  whose  skills 
are  becoming  more  important  as  regular  force  resources  become  more  re- 
strained. 


A desirable  goal  is  to  provide  enlisted  reservists  with  technical,  practical 
and  other  skills  that  will  allow  them  to  perform  satisfactorily  in  specified 
billets.  In  addition,  training  should  contribute  to  retention  by  meeting 
reservists  personal  needs  on  a higher  plane  than  in  the  past. 

Tools  such  as  effective  automated  personnel  accounting  systems,  getting 
reservists  off  to  a good  start  in  their  service  careers,  personalizing 
relationships  with  the  service,  guaranteeing,  when  possible,  equitable 
advancement  opportunities,  a chance  to  earn  retirement  benefits  and 
evaluating  training  programs  with  an  eye  toward  whether  they  contribute 
tc  human  needs,  will  help  keep  our  reserve  forces  strong  and  viable. 
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TABLE  I 

Distribution  of  Enlisted  Personnel  in  the  Coast  Guard  Selected  Reserve, 

Percentage  by  Pay  Grade,  30  July  1973. 

rate  percentage  desired  PERCENTAGE 

E-4  and  Below  53.32  40 

E-5  21.87  30 

E-6  14.97  20 

i 

j E-7  and  Above  9.84  10 

100.00  ion 

Source:  CCGDll(r)  Itr  7430/mmd  dtd  13  Sept  1973 


TABLE  II 


DO  YOU  TUiriC  THE  SERVICEWIDE  EXAM  SYSTEM  IS  FAIR? 


Pay  Grade 

E-3 

E-4 

E-5 

E-6 

E-7 

Combined 

YES 

37.04% 

45.44% 

55.32% 

58.70% 

77.36% 

53.16% 

NO 

18.52% 

25.13% 

21.99% 

14.13% 

13.21% 

20.26% 

NO  RESPONSE 

44.44% 

29.32% 

22.70% 

27.17% 

9.43% 

26.58% 

DID  YOU  COJIPETE  IN  THE  APRIL  1973  SERVICEOTDE  EXAMINATION? 


Pay  Grade 

E-3 

E-4 

E-5 

E-6 

E-7 

Combined 

YES 

5.56% 

7.73% 

10.64% 

11.96% 

18.87% 

10.04% 

NO 

94.44% 

92,15% 

84.40% 

88.04% 

81.13% 

89.96% 

DO  YOU  FEEL  THAT  YOU  HAVE  A REASONABLE  CHANCE  OF  COMPLETING  SUFFICIENT 
TIME  IN  THE  COAST  GUARD  RESERVE  FOR  RETIREMENT? 


Pay  Grade 

E-3 

F-4 

E-5 

E-6 

E-7 

Combined 

YES 

53.70% 

51.84% 

75.18°' 

95.65% 

98.11% 

71.00% 

NO 

46.30% 

44,50% 

19.86% 

3.26% 

1.89% 

26.39% 

NO  RESPONSE 

3.66% 

4.96% 

1.09% 

2.61% 

DO  YOU  FEEL  THAT  YOU  COULD  CONTRIBUTE  MORE  TO  THE  COAST  GUARD  BY  USING 
CIVILIAN  JOB  RELATED  SKILLS  TO  A GREATER  EXTENT? 


Pav  Grade 

E-3 

! E-4 

E-5 

E-6  1 

1 E-7 

Combined 

YES 

62.96% 

56.55% 

66.67% 

65.22%  1 

73.58% 

63.20% 

NO 

37.04% 

39.79% 

31.91% 

30.44% 

26.42% 

34.57% 

NO  RESPONSE 

3.66% 

1.42% 

4.35% 

2.23% 

IF  RESPONSE  TO  THE  ABOVE  QUESTION  WAS  WOULD  YOU 


Pay  Grade 

E-3  1 

E-4 

E-5 

E-6 

1 E-7 

Combined 

PREFER  TO  USE 
CIVILIAN  JOB 
RELATED  SKILS 

48.15% 

39.27% 

35.46% 

65.22% 

i 

73.58% 

37.17% 

PREFER  THE 
VARIETY  I GET 
BY  DOING  SO>fE- 
THING  DIFFERENT 
AS  A RESERVIST 

20,37% 

30.37% 

41.13Z 

30.44% 

26.42% 

36.25% 

NO  RESPONSE 

31.48% 

! 30.37% 

23.41% 

4.35% 

26.58% 

DO  YOU  PLAN  TO 

REENLIST? 

418 

Pay  Grade 

L_ 

E-3  i 

E-4 

E-5 

E-6 

E-7 

Combined 

YES 

11.11% 

19.90% 

35.46% 

71.74% 

98.11% 

40.71% 

NO 

35.19% 

43.46% 

21.28% 

2.17% 

1.89% 

24.72% 

MAYBE 

53.70% 

36.65% 

43.26% 

26.09% 

34.20% 

SOURCE:  USCG  HO  (G-RT)  RECORDS  OF  RESULTS  OF  SW'MER  *73  RESERVIST  QUESTIONNAIRE 
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TABLE  IV 


REENLISTMENT 

POTENTIAL  BY  MISSION 

MISSION 

OBLIGORS 

NON-OBLIGORS 

a. 

Boating  Safety 

65% 

92% 

b. 

Search  & Rescue 

41% 

100% 

c. 

Port  Safety 

27% 

97% 

d. 

Radio  Station,  Base,  Etc. 

25% 

100% 

e. 

Vessel  Augmentation 

6% 

90% 

Source:  Survey  made  by  11th  Coast  Guard  District  (r) 
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TABLE  V 


EVALUATION  (STANDARDS, 
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An  Evaluation  of  Diagnostic  Vocabulary  Testing 
in  Air  Force  Technical  Training 

William  P.  Mockovak,  Lt,  USAF 
AFHKL  Technical  Training  Division  (AFSC) 

Introduction 

Vocabulary  testing  has  tradi tional ly  served  three  general  purposes; 
it  has  been  used  (1)  as  a measure  of  verbal  intelligence,  (2)  as  a mea- 
sure of  total  vocabulary  development,  and  (3)  as  a diagnostic  instrument 
in  examining  different  types  of  word  knowledge.  This  paper  will  be  con- 
cerned only  with  diagnostic  vocabulary  testing  and  its  possible  applica- 
tions as  a measure  of  student  achievement  in  Air  Force  technical  training. 
The  use  of  diagnostic  vocabulary  testing  assumes  the  existence  of 
different  levels  or  types  of  word  knowledge  which  can  be  detected  by 
different  measuring  instruments.  Cronbach  (1942),  for  example,  distin- 
guished between  five  different  levels  of  word  knowledge,  (1)  general ization , 
the  ability  of  the  student  to  define  the  word;  (2)  appl ication , recogni- 
tion of  an  illustration  of  the  word;  (3)  breadth , recall  of  different 
meanings  of  the  word  in  different  contexts;  (4)  precision,  application 
of  the  word  to  all  possible  situations  - even  unfamiliar  ones;  and  (5) 
avai labi 1 i ty , the  student's  use  of  the  word  in  thinking  and  discourse. 
Generally,  diagnostic  vocabulary  testing  has  been  concerned  with  the 
breadth  and  precision  of  word  knowledge  and  the  test  forms  most  commonly 
used  have  been  recall,  multiple  choice,  matching  for  correct  answer, 
synonym,  or  example,  and  the  checklist. 

If  diagnostic  vocabulary  testing  is  to  be  employed  as  a measure 
of  student  proficiency  in  technical  training,  there  are  two  assumptions 
which  must  first  be  satisfied.  The  most  important  of  these  is  that 
tecimicdl  vocabulary  development  must  be  at  least  positively  related 
to  other  measures  of  technical  course  proficiency  or  achievement, 
although  the  magnitude  of  the  relationship  need  not  be  great.  Intuitively, 
tiiis  would  seem  to  be  the  case,  since  in  technical  training,  especially, 
knowledge  of  technical  words  appears  to  be  an  essential  prerequisite 
for  communication  and  skill  acquisition  whether  it  be  through  a 
textbook  or  an  instructor.  Another  assumption  is  that  some  valid 
mecisuriny  instrument  can  be  developed  which  accurately  reflects  the 
level  of  an  individual's  technical  vocabulary  development,  either  in 
at)  absolute  or  diagnostic  sence.  Theoretically,  there  appears  to  be 
additional  justification  for  technical  vocabulary  testing.  Danger 
(1967),  in  discussing  the  relationship  between  vocabulary  and  concept 
development,  states  that,  "the  development  of  vocabulary  is  an 
inextricable  part  of  concept  development."  Within  this  context,  words 
form  the  basis  for  the  classification  of  concepts  since  they  are  the 
"labels"  wiiich  are  cognitively  manipulated  by  individuals  in  thinking 
and  reading.  If,  in  fact,  the  preceding  statement  is  true  and  some 
studies  suggest  that  it  is,  then  there  is  ample  justification  for 
testing  word  knowledge  as  a means  of  measuring  technical  proficiency. 
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For  example,  the  results  of  a study  conducted  by  Randleman  (1963) 
indicated  that  the  size  of  technical  vocabularies  was  positively  related 
to  measures  of  achievement  such  as  IQ  and  scholastic  ability;  however, 
his  results  also  indicated  that  vocabulary  growth  is  attributable  to 
factors  other  than  intellectual  ability. 

The  development  of  valid  test  instruments  to  measure  vocabulary 
development  is  another  problem  in  itself.  Multiple  choice  tests  have 
been  used  most  frequently  because  of  their  ease  in  administration  and 
scoring  but  there  are  distinct  drawbacks  with  this  approach.  For  one 
thing,  studies  have  shown  that  risk-taking  behavior  may  affect  test 
scores  (Slaleter  and  Koehler,  1968;  Ziller,  1957).  Also,  different 
scoring  instructions  have  been  found  to  influence  a student's  overall 
performance.  For  example.  Waters  and  Waters  (1971)  used  a multiple 
choice  format  in  an  effort  to  influence  risk-taking  behavior.  Varying 
tlie  test  instructions  resulted  in  the  following  scoring  procedures; 

(1)  a fractional  amount  was  subtracted  for  a wrong  answer,  (2)  a 
fractional  amount  was  added  for  omitted  items,  and  (3)  a condition 
v;here  there  was  no  reward  and  penalty.  They  concluded  that  if  guessing 
is  to  be  discouraged  on  a multiple  choice  test,  a procedure  for 
rewarding  the  examinee  for  omitted  items  would  be  superior  to  penalizing 
the  examinee  for  wrong  answers.  In  addition  to  scoring  instructions, 
tne  choice  of  which  particular  words  are  to  be  included  in  multiple 
choice  vocabulary  tests  is  also  an  important  consideration  since 
abstract  words,  such  as  adjectives,  adverbs,  and  verbs  have  traditionally 
been  employed  in  vocabulary  tests  as  a means  of  estimating  verbal 
intelligence  (Cureton,  1963).  On  the  other  hand,  a technical  vocabulary 
test  is  interested  in  measuring  achievement,  not  verbal  intelligence; 
therefore,  appropriate  vocabulary  selection  must  be  an  important 
consideration. 

With  the  preceding  considerations  in  mind,  the  use  of  multiple 
choice  testing  to  assess  technical  vocabulary  development  must  be 
approaclied  with  caution.  However,  possibly  the  most  severe  criticism 
of  multiple  choice  testing  when  attempting  to  discriminate  between 
different  levels  of  teclmica!  knowledge  is  that  under  standard  scoring 
procedures,  an  answer  is  either  right  or  wrong,  despite  varying  degrees 
of  student  knowledge.  For  example,  if  four  students  were  asked  to 
select  the  proper  alternative  for  the  word  "polling"  (Question  21, 
Attachment  A),  and  students  A and  B got  the  correct  answer,  but  students 
C and  D did  not,  a supposedly  valid  assumption  would  be  that  students 
A and  B had  equal  states  of  knowledge.  Hov/ever,  it  is  quite  possible 
that  student  A had  no  knowledge  of  the  question  or  alternatives  and 
merely  guessed  correctly;  whereas,  student  B had  sufficient  knowledge 
of  the  term  and  alternatives  to  pick  the  correct  answer.  It  is  also 
feasible  that  student  C had  partial  knowledge  of  the  term  and  was  able 
to  narrow  his  choice  down  to  two  alternatives,  at  which  point  he 
guessed  incorrectly.  Student  D,  on  the  other  hand,  may  have  been 


convinced  that  che  answer  he  picked  was  right  and  all  the  others  were 
wrong.  Obviously,  the  four  students  have  varying  states  of  knowledge 
which  a multiple  choice  test  is  not  sensitive  enough  to  detect. 

Questions  must  be  raised  therefore  about  the  validity  of  the  multiple 
choice  test  as  a sensitive  discriminator  of  student  knowledge. 

In  an  attempt  to  overcome  the  shortcomings  of  traditional  multiple 
choice  testing,  confidence  testing  has  been  proposed  as  a means  of 
determining  a student's  degree  of  confidence  in  various  testing 
situations.  Simply  defined,  "Confidence  testing  is  a method  of  testing 
which  provides  for  weighting,  either  directly,  or  indirectly,  a selected 
alternative  or  alternatives  in  such  a way  as  to  reflect  the  examinee's 
belief  in  the  correctness  of  his  response"  (Echternacht , 1971).  It  is 
hoped,  of  course,  that  the  score  obtained  using  confidence  testing 
would  validly  reflect  varying  degrees  of  knowledge  since  an  important 
component  - the  student's  confidence  in  his  answers  - is  being  included. 
However,  tliere  are  two  important  assumptions  which  must  be  satisfied 
before  confidence  testing  should  be  used;  (1)  it  must  be  assumed  that 
the  examinee  is  interested  in  obtaining  a high  score,  and  (2)  the 
scoring  rule  must  be  known  to  the  examinee.  There  have  been  many 
claims  made  on  the  behalf  of  confidence  testing  such  as  it  increases 
reliability,  it  relieves  student  anxiety  in  testing  situations,  it  is 
a "fairer"  test,  etc.,  but  as  Echternacht  has  stated,  the  ultimate 
value  of  confidence  testing  can  only  be  judged  by  the  gain  in  "ability" 
variation  against  the  gain  in  "error"  variation.  The  increase  in  error 
variation  can  be  attributed  to  such  things  as  (1)  risk-taking  behavior 
patterns  in  expressing  confidence,  (2)  confusion  about  the  scoring 
procedure,  (3)  inability  to  express  accurate  estimates  of  subjective 
probabilities  of  knowledge,  and  (4)  a lack  of  desire  to  perform  well. 
Echternacht  and  Boldt  (1971)  have  reviewed  four  confidence  testing 
procedures  wlii^h  iiave  received  a significant  amount  of  attention. 

They  are,  (1)  Pick  One,  (2)  Distribute  100  Points,  (3)  Sco-Rnle,  and 
(4)  Five  Stars.  These  procedures  differ  primarily  in  the  assignment 
of  weights  to  either  one  or  all  of  the  alternatives;  however,  there 
are  also  significant  differences  in  student/instructor  preferences  and 
administration  and  scoring  time.  It  is  suggested  that  the  interested 
reader  refer  to  the  review  for  a more  thorough  discussion  of  the 
relative  strengths  and  weaknesses  of  the  different  confidence  scoring 
procedures . 

In  light  of  the  previous  studies,  there  appears  to  be  theoretical 
and  empirical  justification  for  using  technical  vocabulary  measures  as 
indicators  of  student  achievement  in  technical  training.  Additional 
empirical  support  is  required  but  the  most  pressing  consideration 
involves  the  development  and  validation  of  appropriate  test  instruments. 
A multiple  choice  test  form  with  a confidence  scoring  procedure  offers 
the  greatest  potential  because  theoretically  it  will  result  in  finer 
discriminations  of  student  knowledges  which  should  have  a significant 
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impact  un  I'anediation  and  the  conduct  of  instruction.  However,  it  is 
also  possible  that  the  traditional  multiple  choice  test  form  with 
"rights  only"  scoring  will  accomplish  the  same  purposes  with  less 
administrative  effort.  In  an  effort  to  clarify  this  issue,  as  well 
as  related  questions,  this  study  attempted  to  accomplish  the  following 
objectives;  (1)  to  determine  the  "feasibility"  of  using  technical 
vocabulary  testing  in  technical  training,  (2)  to  determine  if  Diagnostic 
Vocabulary  Testing  is  a valid  means  of  assessing  technical  course 
proficiency,  (3)  to  compare  confidence  testing  and  multiple  choice 
scoring  formats  in  order  to  determine  if  the  procedures  result  in 
differential  amounts  of  information  concerning  student  proficiency, 
and  (4)  to  determine  the  extent  of  the  relationship  betv/een  reading 
ability,  intelligence,  and  acquisition  of  technical  vocabularies. 


Procedure 


In  the  present  study,  a technical  vocabulary  test  was  developed 
which  employed  the  Pick  One  confidence  testing  procedure.  This  method 
was  chosen  because  it  is  relatively  easy  to  explain  to  students,  it  is 
easy  to  score,  and  testing  time  is  only  slightly  longer  than  traditional 
multiple  choice  (Echternacht  et  al , 1971).  The  scoring  scheme  was 
cnosen  in  sucii  a manner  that  wrong  opinions  confidently  expressed 
incurred  large  penalties,  frank  guesses  and  near  misses  were  only 
mildly  piinished  or  rewarded  if  at  all,  and  confidently  expressed 
correct  opinions  'were  greatly  rewarded.  However,  the  scoring  scheme 
was  also  developed  so  that  a student  would  score  highest  if  he  gave 
an  honest  and  accurate  estimate  of  his  degree  of  certainty.  This 
characteristic  of  the  scoring  scheme  is  termed  "reproducibility"  and 
it  is  discussed  in  greater  detail  by  Boldt  (1971). 

ilie  technical  vocabulary  test  (TVT)  (Attachment  A)  was  developed 
for  the  Inventory  Management  Course  (3ABR64530-1 ) which  is  taught  on 
a resident  basis  at  Lowry  AFB.  This  particular  course  was  selected 
because  it  contained  a sufficient  amount  of  technical  jargon,  yet  the 
vocabulary  was  not  so  difficult  that  someone  who  was  not  an  expert  in 
the  area  could  not  develop  a pool  of  vocabulary  items.  Technical 
vocabulary  was  extracted  from  all  sections  of  the  course  to  develop 
a representative  pool  of  vocabulary  items  and  from  this  pool  35  items 
were  selected  to  represent  the  major  parts  of  the  course.  Each  item 
had  four  al ternati ves , an  example  of  the  answer  sheet  is  presented 
in  Appendix  B. 

The  TVT  was  reviewed  and  corrected  for  technical  accuracy  by 
course  instructors  and  administered  to  40  students  who  had  just 
graduated  from  the  4-block  course  of  study.  Before  administering 
the  test,  the  confidence  scoring  procedure  was  explained  to  the 
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students  and  several  sample  problems  were  worked  through  with  the 
entire  class.  The  weighting  system  used  follows  in  Table  1 (from 
Echternacht  and  Boldt,  1971): 


Table  1 


Credit  if  right 
Penalty  if  wrong 


Degree  of  Confidence 


Very 

Sure  Sure 


Fairly  Not  Very  Not 
Sure  Sure  Sure 


10  9 7 

17  12  5 


3 0 

2 0 


NOTE:  The  instructions  stressed  the  fact  that  a student  would 

score  his  highest  if  he  honestly  expressed  his  confidence. 


in  addition  to  the  TVT,  the  students  were  also  administered  the 
United  States  Armed  Forces  Institute  Achievement  Test  III  (DAT  III), 
Part  I (Vocabulary  Sub-Test).  The  students'  scores  on  this  test  were 
converted  to  estimated  reading  grade  levels  (RGLs)  using  a regression 
equation,  since  there  is  a high  correlation  (.82)  between  reading 
scores  and  vocabulary  scores  on  the  DAT  III. 


Resul ts 


AQE  and  AFQT  data  were  available  for  30  of  the  40  students  who 
took  both  vocabulary  tests.  The  average  scores  were  as  follows 
(standard  deviations  in  parenthesis):  Gen  AQE  = 58.7  (11.0),  Mech 
AQE  = 49.5  (18.7),  Admin  AQE  = 73.3  (12.1),  Elec  AQE  = 59.7  (13.4), 
and  AFQT  = 56.0  (17.5).  The  scores  on  the  TVT  were  obtained  using 
the  confidence  weighting  scheme  and  also  the  traditional  right-or- 
wrony  (rights  only)  scoring  procedure.  It  was  assumed  that  the 
answer  a student  selected  with  the  confidence  testing  instructions 
would  have  been  the  same  answer  selected  with  standard  multiple  choice 
iristr ictions.  The  correlations  of  scores  on  the  TVT  (with  both  methods 
of  scoring)  with  final  course  grades  were  significant  but  the  confidence 
scoring  procedure  did  not  yield  a significantly  higher  correlation 
than  standard  multiple  choice  testing.  In  fact,  confidence  testing 
resulted  in  a slightly  lower  correlation  with  final  course  grades 
than  multiple  choice,  .42  vs  .44  (Table  1).  (A  student's  final  course 
grade  was  calculated  by  averaging  his  end-of-block  test  scores  for 
the  4-bl ock  course. ) 
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Table  2 


Intercorrelations  of  Student  Measures 


UAT  III  (RGL) 

TVT  (Mult  Choice) 
TVT  (Conf  Testing) 


TVT 

(Mult  Choice) 
.46* 


TVT 

(Conf  Testing) 

.53** 

.94** 


Final  Course 
Grade 


.26 

.44* 

.42* 


N = 40 
* p < .05 

**  p < .001 


Nevertheless,  the  significant  correlations  obtained  indicate  that  the 
TVT  is  tapping  a portion  of  course  knowledge. 


Another  question  which  the  data  in  Table  1 partially  answer 
concerns  the  amount  of  additional  predictive  information  obtained 
using  the  confidence  scoring  procedure.  The  two  scoring  procedures 
used  on  the  TVT  correlated  .94  with  each  other,  which  indicates  that 
from  a psychometric  point  of  view,  the  additional  effort  involved  with 
confidence  testing  may  be  unnecessary  because  there  is  no  practical 
difference  in  the  scoring  systems. 


Referring  to  Table  1,  once  again,  it  is  also  interesting  to  note 
the  significant  correlation  between  reading  grade  level  (RGL)  and 
scores  on  the  TVT  using  either  scoring  procedure.  These  data  would 
seem  to  indicate  that  reading  ability  plays  a part  in  the  acquisition 
of  a technical  vocabulary;  however,  v/hen  the  standardized  reading 
scores  were  partialed  out  of  the  TVT  (Mult  Choice)  scores  and  final 
course  grades,  the  TVT  still  correlates.  38  (p  < .05)  with  final 
course  grades  which  indicates  that  the  TVT  is  not  simply  a measure 
of  reading  ability. 


Another  question  which  deserves  attention  involves  the  degree 
of  relationship  between  the  TVT  and  measures  of  aptitude  (AQE)  and 
general  intellectual  ability  (AFQT).  Table  2 presents  the  correlations 
between  individual  AQE  and  AFQT  scores  and  scores  on  the  TVT.  A 
smaller  sample  size  (N  = 30)  was  available  in  this  case  because  of 
missing  student  data. 
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Table  3 

TVT  (Mult  Choice)  TVT  (Conf  Testing) 


Mech  AQE 

.26 

.25 

Gen  AQE 

.27 

.39* 

Admin  AQE 

.22 

.28 

Elec  AQE 

.14 

.18 

AFQT 

-.05 

.05 

N = 30 
* p < .05 

In  addition,  the  lower  sample  size  reduced  the  score  variance  (TVT) 
which  probably  resulted  in  lower  correlations.  The  only  significant 
correlation  obtained  was  between  Gen  AQE  and  TVT  (Conf  Testing).  The 
non-significant  correlation  between  AFQT  and  TVT  is  especially 
interesting  because  it  may  indicate  that  technical  vocabulary 
development  is  independent  of  general  intellectual  ability  as  measured 
by  such  tests  as  the  AFQT.  This  possibility  was  suggested  by  Langer 
(1967)  in  that  he  views  concept  formation  ability  as  being  independent 
of  intelligence.  Actually,  however,  scores  on  technical  vocabulary 
tests  in  general,  and  the  TVT  used  in  this  study,  specifically,  should 
not  correlate  highly  with  "pure"  intelligence  measures  since  most 
technical  vocabulary  consists  of  "concrete"  words,  rather  than 
"abstract"  words  which  have  been  found  to  correlate  highly  with  verbal 
intelligence  (Cureton,  1963). 


Discussion 


In  general,  the  results  of  this  study  seem  to  indicate  that 
technical  vocabulary  testing  is  a feasible  approach  towards  measuring 
a particular  aspect  of  student  achievement  in  Air  Force  technical 
training.  A technical  vocabulary  test  is  easy  to  develop  and  it 
requires  relatively  little  administration  time;  however,  the  question 
arises  of  how  it  can  be  used  in  conjunction  with  standard  student 
achievement  measures  such  as  course  quizzes,  end-of-block  exams,  etc. 
It  would  not  be  desirable  to  use  a TVT  alone  as  an  end-of-block  exam 
because  its  scope  is  not  broad  enough  to  cover  the  diverse  skills 
required  in  most  courses;  however,  it  could  be  used  to  supplement 
block  exams  as  an  additional  measure  of  student  achievement.  Because 
of  its  short  administration  time  (20-30  minutes  for  a 35-item  test), 
however,  a TVT  could  be  used  quite  easily  as  a remedial  aid, 
particularly  if  the  vocabulary  items  are  chosen  with  care  to  reflect 
essential  course  knowledges.  An  instructor  could  refer  quite  readily 
to  such  a test  in  order  to  determine  where  confusions  in  essential 
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concepts  might  be  occurring.  Unfortunately,  the  confidence  testing 
procedure  used  in  this  study  did  not  significantly  increase  the  amount 
of  information  available  from  the  TVT.  Such  information  would  have 
been  useful  in  diagnosing  student  confusions,  but  on  the  basis  of 
other  studies  (Echternacht  et  al,  1971),  it  appears  that  confidence 
testing  is  not  very  feasible  as  a diagnostic  aid  in  technical  training. 
In  the  preceding  study,  confidence  testing  was  employed  with  regular 
course  quizzes  and  different  types  of  remediation  in  two  technical 
courses  in  an  effort  to  determine  if  an  implementation  of  confidence 
testing  procedures  affected  end-of-block  grades,  number  of  remedial 
sessions,  student  and  instructor  attitudes,  and  administration  and 
scoring  time.  The  experimental  design  confounded  the  results  of 
implementing  confidence  testing  but  from  instructor's  responses  and 
attitudes,  it  was  clear  that  they  tended  not  to  use  confidence  testing 
in  planning  remediation.  As  one  instructor  remarked,  "It  reminds  me 
of  using  a bulldozer  to  clear  snow  from  a sidewalk  --  it's  too  good," 
Also,  in  this  study.  Pick  One  confidence  testing  correlated  .92  and 
.89  (compared  to  .94  in  the  present  study)  with  "rights  only"  multiple 
choice  scoring,  so  once  again  there  appears  to  be  little  practical 
difference  in  the  scoring  systems  other  than  changing  the  scale  of 
measurement.  One  feature  of  the  Echternacht  et  al  study  which  does 
deserve  more  attention,  however,  is  the  use  of  special  types  of 
remediation  in  conjunction  with  quizzes  or  TVTs,  since  confidence 
testing  combined  with  special  remediation  did  reduce  the  number  of 
student  remedial  sessions. 

Another  purpose  of  the  present  study  which  deserves  discussion 
involves  the  relationships  between  reading,  aptitude,  and  intelligence 
measures  with  scores  on  the  TVT.  The  fact  that  reading  ability 
correlated  significantly  with  the  TVT  was  expected,  but  the  low  and 
non-significant  correlations  of  AFQT  (Table  3)  with  the  TVT  indicate 
that  technical  vocabulary  acquisition  may  be  a specific  aptitude  in 
addition  to  the  other  aptitude  measures  employed.  It  is  true  that 
the  Gen  AQE  correlated  significantly  with  the  TVT  (Conf  Testing); 
however,  when  the  standardized  reading  scores  (UAT  III)  are  parti aled 
out  of  Gen  AQE  and  the  TVT,  the  correlation  between  Gen  AQE  and  TVT 
(Conf  Testing)  drops  to  a non-significant  .24.  Again,  this  suggests 
that  technical  vocabulary  acquisition  is  a factor  relatively  independent 
of  general  intellectual  abilities.  If  this  hypothesis  can  be 
substantiated  through  further  experimentation  then  the  possibility 
exists  that  training  innovations  can  be  introduced  which  will  help 
individuals  acquire  technical  vocabulary  and  perform  more  effectively 
in  their  career  specialties. 
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Attachment  A 


INVENTORY  MANAGEMENT  TECHNICAL  VOCABULARY  ASSESSMENT 

1.  Issue 

(a)  An  actual  equipment  item  obtained  from  supply 

(b)  A sale  under  the  stock  fund  concept 

(c)  The  procedure  for  obtaining  an  equipment  item  from  supply 

(d)  A computer  process 

2.  Grave  Damage 

(a)  Results  when  destruction  of  equipment  is  over  $5,000. 

(b)  Results  when  unauthorized  disclosure  of  Top  Secret  information. 

(c)  Results  when  an  aircraft  is  grounded  due  to  lack  of  parts. 

(d)  Results  when  a base  is  closed. 

3.  Master  Item 

(a)  An  item  which  must  be  kept  in  a controlled  area 

(b)  An  Item  which  best  meets  the  needs  of  the  AF 

(c)  An  item  composed  of  many  sub-components 

(d)  An  item  used  only  rarely 

4.  AFS 

(a)  Air  Force  Supply  Career  Field 

(b)  Code  for  Army  Federal  Stock  Number 

(c)  Positions  that  require  common  qualification  with  a career  field 

(d)  None  of  the  above 
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5 . SNUO 

(a)  Indicates  the  age  of  an  item 

(b)  A central  file  at  Hq  ATC 

(c)  A central  file  at  Hq  USAF 

(d)  A central  file  at  Hq  AFLC 

6.  Bench  stock 

(a)  Common  items  used  to  repair  end  items 

(b)  rare  items  used  to  repair  end  items 

(c)  Large  storable  items 

(d)  Extremely  large  equipment  pieces 

7.  KIT 

(a)  Parts  used  to  accomplish  a complete  modification  on  an  item  of 
equipment 

4 

(b)  A communication  asking  for  information  from  the  computer 

(c)  A package  of  related  parts 

(d)  Refers  to  a delayed  transaction 

8.  Pre-inventory 

(a)  Preparations  made  for  conducting  physical  counts  of  items 

(b)  A request  for  items 

(c)  A physical  ^ount  of  a certain  item  prior  to  counting  all  items 

(d)  The  name  of  a computer  component 
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9.  Inquiry 

(a)  A request  for  an  item  of  equipment 

(b)  A request  concerning  the  status  of  a piece  of  equipment 

(c)  A remote  terminal 

(d)  A communication  with  the  computer  asking  for  information  of 
same  type 

10.  TCTP 

(a)  An  entry  on  a computer  card 

(b)  Name  for  a computer  file  at  Hq  ATC 

(c)  Orders  from  higher  headquarters  to  perform  a modification 

(d)  Name  for  a computer  file  at  Hq  AFSC 

11.  Excess 

(a)  Temi  applied  to  an  item  due  for  cannibalization 

(b)  An  item  on  hand,  but  which  is  not  required 

(c)  Code  used  to  enter  an  item  into  a computer  file 

(d)  Term  applied  to  delivered  items  that  were  not  ordered 

12.  Demand  level 

(a)  Stock  control  level  based  upon  past  demands 

(b)  Stock  control  level  based  upon  current  demands 

(c)  Stock  control  level  based  upon  future  demands 

(d)  All  of  the  above 

13.  MILSTRIP 

(a)  Military  Storage  Reports 

(b)  A detailed  listing  of  specifications  used  to  describe  items 

(c)  Computer  file 

(d)  Procedures  used  to  requisition  items  from  source  of  supply 


433 


14.  RDO 


1 

1 


(a)  Requisition  directed  order 

(b)  A group  of  regulations  dealing  with  shipments 

(c)  Requested  due-out 

(d)  A cause  for  a directed  shipment 

15.  Quality  Control 

(a)  Examining  documents  to  insure  they  meet  authorized  procedures 

(b)  Examining  equipment  items  to  determine  their  physical  condition 

(c)  Control  of  the  stock  level 

(d)  Hone  of  the  above 

16.  ASC 

(a)  Code  used  to  determine  the  location  of  an  item 

(b)  Code  used  to  indicate  condition  of  an  item 

(c)  Code  used  to  authorize  equipment  to  an  activity 

(d)  Acquisition  supply  control 

17.  CA/CRL 

(a)  Authorized/in-use  detail  record 

(b)  Custodian  Request/Receipt  Listing 

(c)  A listing  of  all  equipment  items  issued  to  a custodian. 

(d)  A listing  of  all  equipment  items  authorized  and  in-use  for  each 
property  custodian. 

18.  Status  notices 

(a)  Notices  used  to  relate  action  taken  on  a requisition 

(b)  Notices  used  to  indicate  the  quality  of  an  item 

(c)  Notices  used  to  indicate  the  quantity  of  an  item  on  hand 

(d)  A communication  with  higher  headquarters 
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19.  Special  Levels 


(a)  Consist  of  minimum,  maximum,  and  fixed  levels 

(b)  Consist  of  predetennined  and  base  initiated  levels 

(c)  Consist  of  additive  and  adjusted  levels 

(d)  All  of  the  above 

20.  MEMI 

(a)  A single  publication  listing  all  equipment  items 

(b)  Master  equipment  managed  items 

(c)  Listing  of  parts  that  go  together  to  make  up  a major  assembly 

(d)  None  of  the  above 

21.  Polling 

(a)  A partial  inventory  of  equipment  items 

(b)  A process  for  communicating  with  the  computer 

(c)  When  the  remote  is  in-line  with  the  computer 

(d)  A clicking  sound 

22.  EMO 

(a)  A button  on  a terminal 

(b)  Code  indicating  the  condition  of  an  item 

(c)  An  authorization  code 

(d)  None  of  the  above 

23.  Recurring  Demand 

(a)  An  item  which  has  been  obtained  from  supply  and  then  returned 

(b)  A special  computer  request  for  information 

(c)  Under  the  stock  fund  concept,  it  refers  to  the  sale  of  an  item 
already  sold  once. 

(d)  A request  for  replacement  of  items  used  in  normal  day-to-day 
operations. 
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24.  Cycle  Inventories 

(a)  Conducted  upon  request 

(b)  Conducted  every  3 months 

(c)  Refers  to  an  item  that  was  once  new,  but  which  has  worn  out. 

(d)  Conducted  when  account  is  transferred. 

25.  LCL 

(a)  Logistics  control 

(b)  A button  on  a remote  keyboard  printer 

(c)  A listing  of  equipment  items 

(d)  None  of  the  above. 

26.  Due-out 

(a)  An  item  ready  for  cannibalization 

(b)  An  item  ready  for  shipment 

(c)  What  occurs  after  due-in 

(d)  None  of  the  above. 

27.  Allowance 

(a)  The  maximum  quantity  of  an  item  required  to  do  a job. 

(b)  Number  of  equipment  items  authorized  a given  organization 

(c)  Refers  to  the  equipment  budget  of  an  organization 

(d)  A special  authorization 

28.  Nonlisted  items 

(a)  Items  which  have  not  yet  been  inventoried 

(b)  A new  shipment  of  equipment  items 

(c)  Items  which  do  not  have  a federal  stock  number 

(d)  Items  which  have  not  been  accounted  for 


436 


29.  IPB 


(a)  Bench  stock 

(b)  A list  of  Items  that  go  together  to  make  a major  assembly 

(c)  A management  office 

(d)  Orders  authorizing  an  equipment  shipment 

30.  Transfer 

(a)  A transaction  in  which  items  are  sent  from  one  base  supply 
warehouse  to  another. 

(b)  Refers  to  interchangeable  items 

(c)  A special  authorization  from  higher  headquarters 

(d)  None  of  the  above 

31 . FSN 

(a)  A special  grouping  of  parts 
(h)  A transaction  identification  code 

(c)  Federal  supply  class 

(d)  An  authorization  code 
32. Inventory  Count  Card 

(a)  An  inventory  overage  document 

(b)  Lists  the  status  of  all  equipment  items  on  inventory 

(c)  Used  to  tag  all  equipment  items  during  inventory 

(d)  Used  to  record  physical  quantity  found  in  the  custodian's  account. 

33.  Suffix  (of  an  AFSC) 

(a)  An  entire  AF  career  field 

(b)  A computer  file 

(c)  Same  as  shredout 

(d)  None  of  the  above 
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34.  Interchangeable  item 

(a)  An  item  which  can  be  used  for  several  purposes 

(b)  An  item  that  can  be  used  in  place  of  another  item  with  little 
or  no  modification. 

(c)  An  item  under  the  stock  fund  concept 

(d)  Same  as  substitution  item 

35.  Nomenclature 

(a)  A full  description  of  an  item 

(b)  Specification  of  an  item  lAW  ATCR  70-23 

(c)  Refers  to  a procedure  for  obtaining  authorized  items 

(d)  Refers  to  a special  type  of  inventory. 
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Attachment  B 


Directions 

Mark  only  one  answer  to  each  question, 
you  are  of  your  answer  by  checking  one 

Indicate  how 
of  the  blocks 
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Not 

Very  Fairly  Very  Not 

Sure  Sure  Sure  Sure  Sure 

[ ] [ ] [ ] [ ] [ ] 

[ ] [][][][] 

[ ] [ ] [ ] [ ] [ ] 

[ ] [ ] [ ] [ ] [ ] 

[ ] [ ] [ ] [ ] C ] 

[ ] [ ] [ ] [ ] [ ] 
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[ ] [ ] [ ] [ ] [ ] 
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Unifora«d  Physician!  witiiout  patiants  i 

1 2 

A Squandering  of  Scare#  Resourcasf 
Qary  B*  Briaaback 

American  Institutaa  for  Research 
Washinjiton,  D.C*  Office 

In  a country  in  which  there  is  a doctor  shortage,  is  it  not  a squandering 
of  scarce  resources  when  nearly  50,000  active  i;hysicians  treat  few  patients  or  | 

no  latients  at  allT  ^ perhaps  not.  Some  non-clinical  functions  may  actually 
require  physicians,  and  some  physicians  may  actually  seek  out  those  functions. 

Optimal  management  of  medical  inanpower  would  include  keeping  to  a minimum  the 
number  of  such  functions  and  t.ie  number  of  physicians  inappropriately  assigned 
to  them. 

In  this  paper  I shall  focus  on  uniformed  phyaiciant  in  the  administre- 
tive  function.  1 have  two  reasons  for  so  limiting  my  topic.  I wish  to  share 
with  you  unpublished  findings  from  an  analysis  I undertook  especially  for 
this  paper  of  some  data  collected  several  years  ago  on  professionals  who  wore 
administrators  in  the  Commiiss ioned  Cor:  s of  the  United  States  Public  Health 
Service  (PIB).  Secondly,  the  : lace  of  physioi-ins  in  health  care  ^id:;iinistra- 
tion  is  becoming  an  increasingly  prominent  and  controversial  issue,  and  I be- 
lieve it  deserves  more  of  our  attention  as  r^seirohers, 

iO  ive  you  some  .dea  of  hov;  much  opinions  differ  an  the  issue,  I should 
like  to  quote  five  statements  seiocted  from  the  literature. 

"It  scares  me  when  I think  of  doctors  ^^etting  into 
management  roles.  They  do  not  have  ti.ne,  and  I 
think  they  are  not  very  good  at  it  anyway. 

Further,  I jo  not  belxove  you  can  even  train  them 
to  be  very  ^ood  at  it  (Guest,  1972,  p.  yH)*" 

You  would  be  mistaken  if  ..wU  thought  the  above  statement  was  made  by  a 
hospital  aaministrotor  wuo  rwu  lad  one  too  m;iny  run-ins  with  the  medical  staff. 
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Th«  parson  who  mads  it  was  a profsssor  of  organizational  bshavior  at  ths  tiao, 
and  ha  was  participating  in  a discussion  of  his  book  chapter  on  *tha  rola  of 
the  doctor  in  institutional  management." 

Consider  a second  quote.  The  president  of  a private  philanthrophy 
recently  saidi 

"The  kind  of  physician  envisioned  in  th is fnaw]  program 
[of  simultaneously  training  medical  resiJrnts  in  manage- 
ment. economics,  and  other  non-clinical  fields]  is  seen 
by  the  Foundation  as  crucial  to  national  progress  in 
health  (Rogers,  1975»  P* 

In  its  re;.ort  to  a former  secretary  of  the  U.  S.  Department  of  Health, 
Education  and  Welfare,  the  Advisory  Coaiaittee  on  Hospital  Effectiveness  made 
the  following  stateaontl 

" appropriate  numbers  of  the  medical  staff  shall  bo  di- 

rsctly  involved  with  the  administration  in  developing  the 
budget  and  operating  plan,  and  in  the  achievement  of  fi- 
nancial and  service  objectives  as  budgeted  and  planned — 

(HEW,  1966,  p.  )." 

The  I'ourth  and  fiftii  views  are  quoted  from  a Federal  agency  recom- 
mendation and  a reply  by  the  Federal  a^-ency  under  investi::ationj 

; "Wo  recommend  [ cons  ideration  bo  given  toj  asii;;ning 

’ nonmedical  personnel  to  administrative  and  mana^e- 

mont  positions  at  the  headquarters  levels  [ and  to 
beginning  a j ro,;ram  to  place  1-SC  officers  with 
master's  degrees  in  hos. ital  administration  into 
positions  as  hospital  cormaanders  (The  Comptroller 
Seneral  of  the  United  States,  1971,  p.  21*)." 

"In  the  military, the  com.^and ing  officer  of  the 

noSi-ital has  oocmand,  military,  judicial,  and 

profess- ijnal  responsibility  as  well  as  admini- 
strative jurisdiction  over  his  s-taff ^nij  must 

be  a physician  (The  Comptroller  General  of  the 
United  States,  1971,  P*  Al)»" 

You  may  recognize  the  fourth  view  as  a recomaendation  from  the  General 
Acco'unting  Office  folio-wing  its  1971  study  of  medical  personnel  in  the  mili- 
■tary,  and  you  may  recall  tnat  the  reply  came  from  one  of  its  tiiree  services. 

I might  add  that  of  the  hor-tatory  literature  on  the  issue  which  I iMive 
reviewed  so  far,  opinion  seems  to  be  running  well  in  favor  of  giving  more 
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phyaii;ian«  more  administrativo  or  managsment  rsaponaibilitiat.  Thara  ia 
ulao  t-iie  view  by  soma  that  the  resolution  of  the  Nation's  health  oare  crisie 
requires  laore,  not  less  medical  leadership* 

Having  already  introduced  to  you  the  specific  subject  of  my  paper, 
its  course  can  be  outlined  thusly.  First,  I shall  review  the  military 
utilization  of  physicians  in  administrative  roles.  Second,  I shall  sum- 
.oariza  ay  review  of  the  literature.  Third,  I shall  summarize  my  own  re- 
-oarch  finain£S  on  the  subject.  Finally,  I shall  0U££est  soma  research  needs. 
ihysician  Ad ainlstrators  in  the  Military 

!-y  review  of  the  military  situation  will  be  brief  and  qualified  because 
I liavo  had  only  an  outside  perspective.  I cannot  even  say  that  I am  an  out- 
sider looking  in  for  I have  never  observed  first-hand  the  work  of  physic iane 
in  the  military.  Moreover,  my  look  has  been  too  quick  for  me  to  ^ain  anytiiinr 
wore  than  a "cneral  imprescion, 

I have  studied  the  staffing  pattern  for  military  pl^'sicians  and  for 
[■iiysicians  in  other  Federal  departments  over  a five-year  period,  I966  to  1972, 
by  examining  end-of-the-yoar  statistical  data  provided  to  mo  by  the  Department 
jf  Survey  Research  of  the  American  Medical  Association.  These  data  are  com- 
, ilod  by  the  Al-iA  from  their  Physicians'  Professional  Activities  questionnaire 
riiich  is  mailed  out  to  all  nl-vaiciuns.  These  data  can  presumably  bo  accepted 
as  reliable  for  I noted  t'nat  the  nuaber  of  physicians  in  command  and  admini- 
strative positions  tnat  was  quoted  in  a June  1975,  Department  of  Defense  nows 
release  was  identical  to  the  number  tabulated  by  the  AMA, 

While  there  are  more  administrative  physicians  in  the  DoD  than  in  any 
other  single  Federal  agency,  the  . eroenta^e  of  the  total  medical  manpower 
iscigned  to  the  administrative  function  the  DoD  is  the  second  lowest  among 
li  of  the  agencies.  Tiie  ailitarp'  staffing  pattern  for  administrative  and 
-;o..  and  positione  iias  been  relatively  level  over  the  five-year  period  studied, 
although  there  ia  a slight  io'vnward  trend  w.hich  began  in  1970  (sec  Figure  1). 
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Xho  five— y®ar  percentAge  of  tiie  medical  nAn]  ower  in  "the  aJiHinist-rativo 

function  i»  slightly  over  seven  and  one-ha.lf  (7»^7'^)«  In  I968,  eij.ht  percent 
of  the  military  physicians  was  in  administrative  and  command  positions,  and 
by  the  end  of  1972  the  percentage  had  dropped  to  about  seven.  There  has 
also  been  a similar  downward  trend  among  the  other  large  Federal  agencies, 

A number  of  blue  ribbon  reviom,  investigations,  and  analyses  imvo  been 

made  over  the  years  of  military  management  of  medical  manpower.  In  I96911 

r 

Lt.  Col.  Gilbert  Jacox  of  the  Medical  Service  Corps  reviewed  IJ  intra  and 
extramural  studies  of  factors  affecting  the  retention  of  medical  officers 
(Jacox,  1969),  He  identified  and  ranked  14  factors  according  to  his  assess- 
ment of  their  negative  effect  on  retention.  Three  of  these  factors  are  more 
relevant  here  and  illustrate  very  well  how  complicated  the  personnel  :anage- 
ment  of  professionals  can  be. 

Consider  the  ir.staoiiity  due  to  frequent  rotations  of  duty  as-ignments. 
Rotation  is  an  equitable  means  of  stall’ing  hardship  posts,  it  is  necessary 
when  medical  officers  are  selected  for  training  programs,  and  it  may  be  an 
effective  way  of  preparing  young  orficers  for  command  positions.  On  the  other 
hand,  rotation  is  the  ^^re^test  deterreiit  to  a .r.edical  career  according  to  Jacox. 
Further,  as  Lando  (197^^)  has  p'ointed  out  in  his  study  for  the  1970  President's 
Conmission  on  an  All-Volunteer  Armed  p'orce,  rotution  disrupts  the  relationship 
between  phj'sician  and  patient  and  the  teamwork  needed  in  uur„ery. 

Another  factor  is  tcie  as i jni:;ant  of  medical  ofi  icers  to  administrative 
and  com:.and  positions.  Three  of  the  IJ  studies  reviewed  by  Jacox  concluded 
that  tills  factor  vas  a major  one  in  the  turnover  of  ..edlcal  officers,  and  Jacox 
uLmself  rarnced  this  factor  eighth  among  the  14  in  contr ibutiiig  to  turnover, 
Jacox  noted  the  ai^arent  ;ara-ox  of  ti.e  medical  leiier'-hip  uryiing  for  the 
tieceOiity  of  1 .ly-. _cian  ad  inistritors  and  com  ..anders  •vnile  at  the  same  time 
separating  medical  officers  were  cluing  t.ieir  dread  of  such  assignments  as  a 
motivation  for  leaving  the  military. 
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Although  not  re^iirdod  by  Jaoox  a*  having  too  significant  an  impact  upon 
retention,  a third  factor  which  is  relevant  hero  is  that  of  the  professional 
and  Departmental  leadership  of  military  physicians.  Two  of  the  IJ  studios 
Jacox  reviewed  concluded  tiiat  lack  of  professional  leadr.rsnip  was  a major 
-obati-cle  to  retention  of  medical  officers.  It  '<ra.s  further  concluded  that 
reliance  upon  the  draft  as  a recruitment  source  of  physicians  permitted  a more 
casual  management  of  pliysicians  and  medical  programs.  If  tills  conclusion  ia 
valid,  we  sh.ould  uxi  ect  itanager.ent  style  to  clianye  with  the  recent  abolition 
of  the  draft.  In  any  event,  we  may  continue  to  have  what  I believe  is  still 
another  paradox.  There  is  uhe  general  tuinicing  perhaps  that  one  of  the  reasons 
it  is  necessary'  I'or  a j.ealfh  administrator  or  manager  to  be  a physician  is  the 
professional  resject  he  automatically  enjoys  frooi  the  physicians  under  him. 

Unloib  the  jnysician  is  a top  .lotch  administrator,  or  iias  a strong  admini- 
strative deputy  or  aid,  :ie  is  under  the  strain  of  attempting  to  achieve  reepect 
as  a loader  as  well,  fet  if  he  needs  a strong  deputy,  then  he  is  vulnerable 
to  criticism  for  the  very  problem  we  began  v-ith,  the  staffing  of  administrative 
and  cooujiand  oaitions  w'ith  inysiciuns. 

This  j roblem  was  in'/esti  ated  in  lp71  by  the  General  Accounting  Office 
in  the  study  (The  Comptroller  General  of  the  United  States,  lp71)  mentioned  abo’,'9. 
Ill  this  study  a number  of  billet  acjcriitions  of  ad._inistra t^ve  and  staff  medi- 
cal officers'  positions  were  examined,  and  the  officers  were  asked  if  their 
work  required  a medical  liack  round,  A^l  but  a I’ew  officers  believed  their 
::.eaical  bad:  round  '•■'ac  necescarp'.  PJeverthcleas  the  invtcti^ -tcrc  found  what 
t .cy  believed  \;ere  a number  of  uei-tionable  staffing,  practices,  such  as  admini- 
atr  1 tor-dai  uty  ^airs  in  •..'hioh  botm  incumbents  were  dnysicians.  T;ie  investi- 
r, a tors  concluded  that  officers  ot.ior  than  physicians  could  be  placed  in  charge 
of  medical  facilities  and  in  avi,,.ini.itrative  and  raanarcmsnt  j'os’.tion:  at  the 
je,  artiaentul  level. 
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Ono  year  a^o,  the  Secretary  of  uefense  and  the  Secretary  of  Health, 
Education  and  \yolfare  is-ued  a joint  rs;ort  (DoD-HTrf,  1P72)  to  the  President 
and  Congress  on  v'aya  of  rcducinj  the  reliance  on  nedical  officers  while  alao 
aeetinc  the  Jiedical  nev'ds  of  the  anaed  forces.  The  report  concluded  that  a 
reduction  DoO  manpower  requireaonts  for  medical  officers  \.’as  possible  (the  pro- 
jected reduction  amount' d to  about  a 20^o  shavinj  from  the  authorized  strength 
of  June  1972).  The  report  summarized  five  ways  in  wnich  the  reduction 

could  be  ef-octed.  While  substitution  of  non  [hysicians  in  the  administrative 
function  was  not  one  of  the  five  xeciianisms  mentioned,  the  re.ort  did  note  that 
an  experiaental'  test  of  the  concept  of  small  treatment  facilities  under  the 
coni:  and  of  non  ; hysicians  was  then  bein^  made. 

Apparently  this  tost  v;as  at  least  partially  ef.ective.  As  you  may  know, 
the  Je.uty  Decretory  of  Deimoe  announced  on  June  1 of  this  year  that  there 
•rill  be  a _ radual  imple.r.e'rtation  of  a ;olioy  •.■  '.ich  ;er:nits  the  pl'iciny  of 
qualified  off  .cere  without  consideration  oi'  t-.eir  bade  ;.r  ofesc  ional  s-tatus 
in  positions  of  com.r.ani  or  acidnistrstive  dir-:ctlon  oi’  military  nealth  facilities. 
Thus  such  ;Ositions  will  no  lor.-er  be  filled  only  by  priys  ic  ians . Further,  phy- 
tioiens  perionjiinp;  ed...  inictr tive  duties  in  •- to  . f level  positions  ••■ill  be  re- 
placed 'ey  other  health  profes  ionals  as  much  as  .ossible. 

Jiven  tr.e  re;  uted  scarcity  of  .oisns,  t..eir  utilization  -.iithin  the  DoD 

in  •.'.•ori.r  not  aircctly  related  o ;atient  crorc  sh'oulJ  continually  „e  open  to  ob- 
ject ■.ve  review.  As  yo'u  ...ay  Imow,  tlie  General  jkccountin"  Office  is  ;reparinr 
to  launch  another  snu  broader  inveoti  »tion  into  f.te  .ounap^oxent  of  health  pro- 
I'js-ionals  in  the  ..ilitory.  The  'Office  of  Msna 'Cr.ent  and  2ud  et,  I am  also 
told,  -/ill  -oon  initiate  a laryc-scale  study  of  fnis  peneral  subject. 

■..’hat  pue..ti-rs  could  v/e  ask  Juraelves  today  about  hov/  to  'oana,;e  a scarce 
•rce  ■■  .-h  !as  ^,ro...in,  and  c -Oi-.pet in,  dc'ands  ^ Isotd  u]  on  it?  it  seems  a 
- : r ...c  ue^ti.-ns  w'juld  be  the'oe:  b ?es  tu-a  work  Qoooiutcly  require  a 

. ( I.  -r.,  ^'iio  '.r-;  the  ...o.;t  ■.puuliilcd  . . i.ians  lor  the  -work? 
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Should  thoro  be  a definite  administrative  career  path  from  entrance  level 
to  top  coGUjand?  If  yes,  what  effect  does  this  have  on  recruitment  and  sub»<e- 
quent  retention  of  physic ianaf 

I do  not  pretend  to  have  the  answers  to  tiiese  and  similar  questions. 

But  before  proceeding  to  the  next  section  oi  tais  paper,  i siiouia  lilca  to 
offer  tn«s  following  general  impression  of  mine.  It  appears  to  me  the  DoD  may 
be  caught  up  in  a dilemma  in  a period  of  uncertainty.  On  the  one  hand,  we  have 
v/itnessed  a alow  decline  in  the  percentage  of  medical  oflicera  in  administrative 
rolea  over  the  last  few  years.  Ihis  decline  is  perhaps  in  reaponse  to  the  very 
invaatiqatlona  and  in-houae  analyaeo  which  I have  rsviev/ed  here  today.  The  de- 
cline lu3is  just  become  stated  Departmental  policy,  and  it  could  very  well  be  an 
ab- )lute  necessity  for  an  all-vol'antoor  armed  force.  On  the  other  hand,  however, 
we  are  also  witneseing  growing  opinion  in  the  health  field  in  favor  of  phy- 
- .oianc  lecotiiin_;  more  active  in  the  management  of  health  care  facilities, 

What  is  the  solution?  There  is  scarcely  any  evidence,  as  shall  see  in  a 
moment,  to  uupt  ort  or  refute  either  argument.  It  is  difl'ioult  to  be  very  cer-  ! 

tain  about  management  decisions  in  the  face  of  such  uncertainty.  i 

Beviev.'  of  the  Literature  ] 

As  far  as  I can  tell  i .'om  my  review  of  the  literature  on  tiie  issue  of  | 

,n^-siciana  in  ad  .inistr  itive  roles,  tnere  lias  been  quite  a bit  ol  opinion 
ar.d  no  research.  Almost  all  of  t..e  opinion,  incidentally,  appears  in  the  lit- 
erature of  t a nealth  I’ield,  such  as  the  journal  of  the  A.'.:erioan  hospital  I 

Asaocljtion,  hos ;•  Ital . A I^IULARS  search  which  I requested  from  t!ie  national  ' 

Library  of  I'.eJicine  of  articles  indexed  by  lijlDIClTo  netted  63  oitafions,  not 
one  of  vin^ch  was  a report  of  any  research  study.  I reviewed  five  and  one-lialf 
years  of  psyoholo  ioal  Abstracts  covering  the  period  of  January,  1P68,  to 
July,  i?7y»  h'ot  one  of  the  approximately  I'^CO  citations  indexed  under  either 
tile  subject  heading  of  “iGed  ic  ine"  or  " phj/a  i;  ian"  re;  resented  a research  study 

h 

of  the  is-ue.  i:or  uid  I fina  any  research  at  idy  out  of  more  than  700  citations 
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during  that  iame  period  under  the  heading  of  "personnel  in  the  military." 

I also  requested  a Defense  Documentation  Center  search  for  any  published  or 
unpublished  documents  on  the  issue.  This  search  yielded  nearly  I9O  ci- 
tations, and  not  one  was  a report  of  any  research  on  or  an  analysis  of 
the  issue. 

Other  than  the  liklihood  tliat  onoerteinlp  lules  over  the  issue,  vrhat  else 
can  be  concluded  from  my  literature  review?  Firi-t,  since  a considerable 
body  of  knowledge  has  been  accumulating  on  leadersnip  and  managerial  ef- 
fectiveness (there  are,  for  example,  nearly  oOO  ref-rences  listed  in  the 
highly  regarded  book  by  Campbell,  Dur.nette,  Lawler  4 Weiok,  1970>  on  mana- 
gerial behavior),  it  is  the  medical  profession  and  health  care  establ iohaent, 
and  not  the  adninistr.Mtive  f'unotion  itself,  obviously,  where  researchers  have 
failed  to  make  an  entry.  Why  is  this  so? 

There  is  the  belief  that  physicians  are  unwilling  subjects  because  they 
have  too  little  time  to  spare  and/or  they  see  research  as  a threat  to  the 
status  quo.  I question  how  close  to  the  truth  t.'.is  belief  is.  Several  thous- 
and physicians  in  a large  bureaucracy  have  willingly  participated  in  my  own 
research.  I suspect  the  medical  profession  and  health  care  establishment 
may  be  aomev/hat  sensitive  about  the  isf:ue,  but  faoy  are  also  perceptive 
enough  to  reoogniue  t;mt  resistance  to  rowing  deuends  for  .;ublic  inquiry 
vjould  be  self  defeating.  Uniortunatoly , industrial  and  organizational  psy- 
chology, one  of  the  disciplines  capable  of  as  iating  the  medical  profession 
and  health  cart  eststl iahiient  to  txu.oine  the  issue  objectively,  apparently 
lias  eitfisr  i;  nored  or  be.^n  ui-.u-oare  ol  the  is  ue  (I  should  add  t.iat  psy- 
cholo  istc  in  ^er.sral  have  had  a myo^-ic  viev;  of’  health  and  health  care# 
one  of  my  ;pOals  on  other  occasions  rjaa  been  to  promote  a broader  view  of  the 
problems  wiiioh  pervade  the  liealth  care  Indus-*  -y-e , bru-nback,  197-). 

My  second  conclusion  is  tliat  until  vie  iiavc  the  research  findings  we 
need  on  2 ecific  questions  about  p ic ians  as  mana  ers,  wt  should  begin 


r " ^ 

trying  to  draw  parallels  amonf  administrative  positions  in  asdicins  and  bsalth 
care  and  those  positions  in  the  larger  segment  of  the  managerial  world  where 
so  much  knowledge  already  exists* 

Unpublished  Research  on  PHS  Physician  Administrators 

I wish  now  to  turn  briefly  to  ny  own  unpublished  research  on  physician 
administrators  in  the  Commissioned  Corps  of  the  PIS*  This  research  should  bo 
regarded  as  strictly  exploratory  because  the  bulk  of  data  I have  analyzed  was 
a by-product  of  a multi-phased  research  program  which  I designed  more  than 
five  years  ago  for  a purpose  other  than  testing  specific  hypotheses  about 
physician  adcinistrJitors,  Therefore,  the  findings  I am  pre;.enting  here  are 
a result  of  post  hoc  analyses  of  a potpourri  of  data* 

The  research  program  was  an  effort  to  develop  and  validate  a job-content 
based  performance  appraisal  system  for  commissioned  officers  in  the  p®* 

The  program  and  its  major  results  hmve  been  reported  elsewhere  (Brumback  L 
Vincent,  1970a,  1970b;  Srumback  i-  Howell,  1971a,  l971b,  1972)*^  To  give 
you  some  background  information,  -"’he  first  phase  of  the  program  involved 
, a factor  analysis  and  then  a hierarchical  cluster  analysis  (a  la  Christal,  19^2) 

of  a stands ruizad  position  inventory  which  had  been  constructed  and  admini- 
stered to  about  5OGO  P®  of-^icers.  The  second  pliase  involved  collecting- 
over  2,500  critical  inciaant  re-orts  of  job  performance  from  -.bout  5^0  officers. 

Thie  third  and  final  phase  was  a validation  of  an  experimental  efficiency  re- 
port against  a criterion  of  averaged  work  associate  ratings  on  a sample  of 
about  1200  off  icers  , 

I hiave  explored  five  questions  using  ^ome  of  the  by-product  data  from 
the  above  program.  For  each  question,  the  independent  variable  is  the  typo 
of  position  held  by  the  ihysician  or  his  comparison  ^;roup*  The  two  types  of 
positions  of  concern  here  are  administr.ative  positions  and  patient  care  positions. 

Position  type  was  determined  by  the  hierarchical  clustering  technique  menticned 
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aboT«,  ® This  taohniqu®  had  produced  quite  diatin£uiahablo  position  types 
or  Job  families.  For  example,  there  were  no  patient  care  duties  among  the 
top  5056  of  the  duties  which  accounted  for  the  group  job  description  of  phy- 
sicians in  the  administrative  job  I'amily.  I have  limited  my  analyses  to 
Director  grade  officers  within  each  of  the  job  families  because  officers 
at  this  next  to  the  highest  level  are  careerists.  Further,  there  ware  too 
few  Sxirgeon  General  grades  for  statistical  comparisons, 

l.^y  first  question  «s  tnis  j v.’ere  pl:^.'sicians  who  became  career  admini- 
strators less  clinically  proficient  at  the  time  of  entry  into  the  P®  than 
v/ere  physicians  who  remained  as  clinicians  throughout  their  FH3  career? 

I was  able  to  trace  back  to  the  entrance  professional  examination  records 
for  a physician  sample  of  career  administrators  (1,'  - 16)  and  a matched  ply- 
sioian  sample  (K  = 16)  01  career  cxin-w.ji.ne  7 At  inat  time,  ptx/siwians  v;ere 
selected  into  the  Corps  on  the  tar, i.  -of  u final  veirhted  avcrap;e  of  tlieir 
v/ritton  and  oral  examination  scores  and  file  or  profesrional  background  re- 
viev;  scores  (see  re^.'man,  l95i»  foi"  ^ description  of  the  c ficer  selection 
i rograun  of  the  P®).  I computed  the  mean  of  the  final  standard  selection 
aoores  I'or  each  group  and  found  tint  tiiey  v;ere  identical  (a  mean  score  of 
for  each  group).  Parhuis  contrary  to  onet  expectation,  career  ad_ini- 
ctr-tors  \;ere  not  judged  anp-  less  qualified  as  clinicians  tican  were  the  career 
clinicians  at  tho  time  of  er.trp'’  into  the  F®. 

I-!y  next  question  vreis  t at  the  time  of  entry  into  the  F®,  did  career 
administrators  have  initiallp'  different  career  intentions  and  work  preferences 
than  did  the  career  clinicians? 

I was  able  to  retrieve  and  examine  for  tvfo  matched  samples  of  icians 

(•'  “ ‘’6)  t.ieir  reL..o;iae3  to  a Service  Evaluation  Questionnaire  (3EQ)  which  had 

ieon  administered  uc  an  attitude  survey  to  officers  cevoral  years  prior  to 

o\m  researol  program  (-ce  Mc-.'ell,  3-'u_iback,  ;To->-..;an  L Rir-o,  19^6,  for  a de- 

tt 

scription  of  the  dEQ  research).  I found  no  differer.ce  between  the  two  groups 
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in  thair  initial  career  intention*  (Chi  Square  ■ l*lf  Fo^*  botli 

career  administrators  and  career  clinician*,  only  slightly  over  10!^  of  them 
stated  they  initially  had  planned  to  leave  the  Pi6  after  fulfilling  their 
;,iilitary  obligation*.  The  liajority  of  the  physician*  were  either  open- 
minded,  favorably  disposed,  or  co.iir.itted  to  a career  at  the  time  of  entry. 

I did  find  a significance  difl'er!:r.ce  (Chi  Square  = J6.7,  p^.OOl)  between 
the  two  jroop*  in  their  initial  woric  preferences.  Whereas  most  of  the 

career  clinicians  preferred  patient  care  work  at  the  outset,  career  admini- 
strators were  about  evenly  split  between  preferring  patient  care  work  and 
various  other  kinds  of  initial  as.-^ i;jn.aents , 

Thirdly,  I v/as  curiouc  as  to  whether  career  administrators  are  less 
-atiefied  with  their  careers,  w iich  s .'ce  .si^ht  ro._ard  as  :arginal  role 
'laying,  than  are  career  clinicians.  I found  no  i;.ean  iifj’erence  (t  = 0.7, 
p^.d5)  botv.-een  overall  3Zi  scores  for  the  two  ,_roups.  Both  rou.  s v.'ere 
equally  satisfied  v.'ith  the  careers  they  liad  .-.lade. 

Fourthly,  I v.anted  to  knov?  whether  hi  Iv-level  physician  administrators 

in  the  PiS  become  administrator?;  over  nl, ';t  or  have  :iad  .revlous  aatini- 

ntrative  nssi  nnonts.  result.';  i.viioate  the  latter;  ic l.anB  who  be- 

esme  hi  ,h-level  adisr.-iistrs  tore.  ;-ad  worked  t.’.eir  v.uy  up  tnrough  a pro_ression 

of  incress in  ly  res;onsibl-  admin is trative  jositions.  puysicians  in  the  ad- 

...inistr -itive  job  family  had  served  an  avers  ye  of  ab.out  nine  and  one-half  years 

in  uaministrutive  assignments  before  reaohinr  tneir  Jircctor  grade  .osts. 

Moreover,  come  cf  these  ihysioiana  had  acquired  admir.istrMtor  training  through 

rad'uate  aeyree  programs  in  public  health.^  In  addition  .0  j.ro^reariny  throu,„h 

a career  1 ath  of  administrative  lositions,  the  i;i  ,h-level  administrators  ty;  i- 

o:*lly  had  rotated  throu;,h  a variety  of  non-adminiatrative  positions  earlier 

in  dhair  career,  prior  to  their  succession  of  ad.r.lniatra tive  assignments, 

these  physicians  ;vid  served  an  average  of  about  10  years  in  non-adm inistrative 
10 

, os iti  ins . 
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>*y  last  quastion  was  this  i how  affaotivaly  do  physicians  perl'ora  as 
administrators  comparad  to  other  professionals  in  administrative  positions? 
This  could  bo  called  the  |64  question. 

If  physicians  do  not  make  very  -ood  administrators,  as  the  professor  of 
or'^inizational  behavior  mentioned  earlier  fears,  my  findings  indicate  at  least 
timt  physicians  do  not  appear  to  be  any  ’'.oroo  administrator  a tlvxn  other  pro- 
feacionala  in  administrative  positions.  Although  more  critical  incidents 
were  reported  on  physician  administrators  than  would  be  expected  according 
to  their  reprecentution  In  the  PHS  (Chi  Square  = 21. 4o,  p^.OOl),  there  was 
no  £.•  i _nif leant  difference  betv;een  the  number  of  re^-orted  ne -ative  incidents 
of  adminiatrotive  performance  of  | Ivcicians  and  the  nuicber  of  negative  inci- 
dantc  reported  on  other  non-physician  admin  is  trs  tore  (Ohi  Square  = J.77»  P 
Further,  analysis  of  variance  showed  no  significant  difference  in  the  mean 
work  associate  ratings  of  ef fsetivsness  of  auministrd tive  ^erl or;.iance  among 
ihysicians  and  10  other  profes . i .-■■.lal  grou;  s (F  = 1»5^»  P>.05)»  Considering  the 
fact  that  the  ihysicians  tended  to  occupy  the  ^ore  rc-;o.;sible  administrative 
positions,  even  with  ^ride  level  constant,  t .an  aid  the  other  professional 
groups,  physicians  would  appear  to  iiave  mors  t.ian  held  t'.isir  own  in  the  dis- 
charge of  tneir  reapionsibiiities.^^ 

I should  like  to  draw  some  tentative  conclusions  from  ay  research. 

First,  phj’siclan  administrators  do  not  at  the  beginning  of  their  career 
avoid  or  leave  clinical  medicine  because  they  lack  clinical  proficiency. 

Indeed  they  wore  Judged  no  less  rroficient  as  clinicianc  than  v/ere  the 
career  clinicians.  Second,  some  physician  aiministra tors  prefer  nour- 
cllnioal  work  at  the  begin.. ing  of  their  careor  even  tnough  they  probably 
would  have  been  proficient  clinicians  in  practice.  I suppose  the  reasons 
for  this  are  to  be  found  in  the  interests,  personality  and  biographical 
characteristics  of  such  physicians.  Third,  phi’^'i^ian  administrators  are 
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not  frustrated  actors  playing  in  marginal  rolas.  Ih#  physician  admini- 


strators I studied  appeared  to  enjoy  their  work  and  career.  Fourth,  phy- 
sicians do  not  make  bad  administrators.  Fifth,  physicians  can  be  prepared 
and  trained  to  be  good  administrators. 

Further  Research  Needs 

The  utilization  of  physicians  in  administrative  roles  is  an  important 
and  controversial  issue  •,;hich  begs  for  more  research  attention.  I should 
think  that  the  $123  -uestion  that  needs  to  be  ansvfered  is  whether  or  which 
health  administration  and  manage;.:ent  positions  absolutely  require  a medical 
education.  Do  soma  of  --i-.ese  ,ositions  really  require  physicians,  or  ia  it 
more  a case  of  some  ;hy3icijns  really  requiring  the  positions?  What  is 
needed  is  a systematic  classification  or  ta.xonc.^  of  all  hi,. her  level 
positions  in  the  health  and  healtli  care  field  across  or,;ani/-ations  and  fur.ctions. 
This  classification  ohould  be  determined  by  the  critical  knowledge  require- 
ments of  policy  and  decision  making  nd  the  critical  interpersonal  skills 
needed.  If  policies,  decisions  and  leadership  Jo  not  demand  a medical  edu- 
cation, then  a ni,_h  level  administr itive  or  management  position  s.iould  not 
be  filled  by  a ;h;'sician.  The  classification  should  also  distinguish  those 
positions  in  which  melicol  leadership  is  not  critical,  but  in  which  close 
medical  conaultadion  to  tho  leader  is. 

I wonder  if  the  state-of-the-art  of  our  job  analysis  inet -lodoloyy  is  uu 
to  the  cliallenyo.  Obviously  a physician  incumbent's  rationale  for  nia  assi  n- 
_.ent  is  not  - oln~  to  be  a sufficiently  vs  lid  justificapion  for  requiring  a 
medical  ecucation.  Comparisons  oi'  staffing  patterns  am  mg  organizational 
units  may  reveal  incona is  tens ies , but  these  may  be  due  just  as  much  to  dif- 
ferences in  actual  r.;  ..uira.iients  a.,  to  arbitrai-q'  practices,.  One  needs  more 
data  than  tao  .j.ore  fact  in  two  .or  ..ore  similar  units  one  is  headed  by  a 
..hysioiun  with  a c-rtsin  position  title  one  the  other  is  headed  by  a non 
.hp'sioian  'ith  the  s.^me  title.  One  v;ould  also  need  ;;ore  than  the  standard 
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poBition  inventory  analyei*  to  determina  critical  knowledge  requireaenta. 

For  the  mowont  I frankly  am  not  sure  how  I would  proceed  aethodolotJ.aally. 

I Buppose  a poasible  approach  would  be  to  have  both  a job  analyst  and  a aub- 
ject  matter  export  thoroughly  analyze  the  content  of  each  critical  decision 
and  policy  and  each  leadership  act  for  a sample  of  administrative  positions 
hold  by  phyaiciana.  T’lie  infor-a tion  coula  then  be  useu  to  ouiiu  ai*  in- 
oaaket  toa^,  If  non  phj'sicians  could  reproduce  the  policies,  deci'-iona  and 
leadership  acts  of  the  physiciuna,  there  would  bo  sore  reason  to  believe  that 
a medical  education  is  a critical  requirement. 

Another  importa.it  area  for  research,  aa.umini  ti.ere  are  indeed  a number 
of  s—_*nistrutive  positions  '■■hlch  require  a ::edical  -.ducation,  is  the  early 
ideiitif ication  and  devtlo. ..lent  of  -ana^erial  talent  ason-  physiciana  or  even 
a .on,^  .oeJioil  .tudsnts.  What  arc  .he  Individual  differences  that  can  be 
measured  and  used  to  predict  subsequent  specialization  in  health  adminiatrationt 
Bio£;rai.hical  data  as  ^rcaictors  siiould  e ex.lored  as  well  a*  interest  a.;d 
i-ersonality  measures,  .Howell  and  h'e-.-.'.an  (19^7}  reported  on  a concurrent 
validity  study  of  a combination  biographical  and  personality  inventory, 
i.iese  lesearohers  n'ore  able  .0  .Ifftr  ntiate  .':s  .ost  clearly  betv;een  phy- 
sicians i.n  .-.'.'■■dical  care  and  ^ublic  i.ral  ih-a.i, , iaiistro  ti.'n  on  the  basis  of 
a ns ’■.•era  to  the  invintorj-, 

Anotner  crucial  'n  Is  ••  ;on  and  ho-.;  duria,^  the  , roceSd  of  medical 

education  or  ; rofss sional  ;.rsctict  should  j hp'siciana  learn  aJminiutra tive 
and  .r.sna  yiri.jl  skills?  A nev;  ^Tojr'xv.  for  yivin^  oiana^^e.r.ent  traininjj  to 
a.edioal  residents  vf-a.-.  cit.d  above.  What  should  be  the  content  of  such  a 
•ro^.,raEf  As  Campbell  et  al.  (I97C)  have  obaerv.d,  there  ;ias  been  little 
research  on  training  content  us  o^-.osed  to  toenniquea  in  jianaipe.xont  de- 
vclo,ment  and  triinin^  ^.roj^rar..s , Longitudinal  research  is  also  needed  on 
the  kinds  of  career  ey,  er  .a.-.ce-  •.'hich  lead  to  succes.,ful  ad.oinistr  - tive 
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^P»p®r  presentsd  at  the  1975  Military  Testing  Aaaooiation  Conference, 

San  Antonio,  Texas,  28  October  — 2 November  1975» 
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at  various  stages  of  the  research  and  preparation  of  this  paper.  I wish  to 
thank  in  particular  two  staff  members  of  American  Institutes  for  Research, 

Mrs.  Roberta  Marcum  for  assistance  in  the  data  analyses  and  Mrs.  Lily  Qriner 
for  library  support  in  the  literature  review.  I also  acknowledre  with  thanks 
Dr.  Jimmy  L.  Hatfield  of  AIR  for  his  consultation  on  information  sources. 

I an  also  deeply  r;rataful  to  Miss  Gene  Roback  and  her  staff  of  the  Department 
of  Survey  Research  of  the  Anerican  Medical  Association  for  providing  me  with 
statistical  data  on  the  distribution  of  physicians  in  the  United  States  over 
a five-year  period.  Thanks  are  due  several  Department  of  Defense  individuals, 
Mr.  Mark  Bie-rle,  Ur.  Ralph  Cantor,  Col.  Vfn.  U.  Augerson  and  Col.  Billy  F. 
Simmons  for  their  helpful  information  obtained  via  telooons.  In  addition. 

Col.  Sixamons  was  especially  helpful  in  providing  me  with  doovnented  materials. 
Thanks  are  also  due  several  General  Accounting  Office  individuals,  Mr.  Charles 
Collins,  Mr.  Gene  Hamnel  and  Mr.  Max  Stettner  for  their  helpfal  information. 
Impossible  thouph  it  would  have  been  to  complete  the  research  and  paper  viith 
out  t':e  contribution.';  of  the  above  named  persons,  they  are  ir  no  m.y  resj  onsi- 
ble  for  vmatever  imperfections  may  appear  in  this  presentation. 

^'his  rounded  figure  indicates  the  approximate  number  of  active  and  classi- 
fiable physicians  in  the  United  States  whose  primary  professional  activity  is 
not  directly  involved  in  patient  care.  The  figure  is  based  on  data  compiled 
from  a computer  listin-  furni'shed  to  mo  by  the  Dcpart.uent  of  Survey  Research 
of  the  A.rerioan  Medical  Association.  The  AMA  criterion  of  primary  activity 
is  the  largest  percentage  of  time  spent  by  the  p!\,’sicia;  in  a single  activity. 
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Th«  AMA  dofinss  direct  involvmant  in  patient  care  as  the  seeing  of  patients, 
but  also  Includes  the  patient  services  of  such  ph:,'siciana  as  pathologists 
and  radiologists, 

^he  total  count  of  1^0  citations  is  somewhat  inflated  since  there  are 
soiae  overlapping  citations  under  the  two  headings  of  "physician"  and  "medicine." 

Reprints  of  published  work  from  this  research  program  or  from  related 
efforts  are  available  from  the  writer  upon  request. 

^Position  type  was  identified  from  the  hierarchical  clustering  technique 
for  all  but  the  last  question.  In  that  instance,  administrative  physicians 
;;ere  identified  by  their  scores  on  v;ork  associates'  ratings  of  functional 
performance  and  by  type  of  critical  incident  report. 

"^Tha  two  samples  were  matched  on  entry-on-duty  date. 

^The  two  samples  were  matched  on  tenure. 

°Pifty-five  percent  of  the  ph;,-3ician  administrators  had  graduate  degrees. 

It  is  not  irnown  what  percentage  of  tiiis  group  had  master  or  doctoral  degrees 
in  public  health  administration. 

^®It  is  not  knovm  what  the  assignment  ]3attsrns  were  for  the  physicians 
before  they  becume  administrators.  prei'U.;ably  some  of  them  began  their  P® 
enrloyment  as  clinicians. 

^^Piiysioians  supervised  si__nifioantly  more  staff  and  controlled  larger 
budgets  t;ian  did  the  other  > rcfess  ional  groups. 
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1969 


1970 


1971 


1972 


Year 

Pit,.  1.  Percsntage  of  active  and  classifiable  phyoiciane  in  the  USA  whose 
priaaary  professional  activity  is  adisiniatrative  work.  (Based  on  data  com- 
piled by  the  Department  of  Survey  Research  of  the  Anericon  I'.edioal  Association). 
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AN  EXPERIMENTAL  IMPLEMENTATION 
OF 

COMPUTER  ASSISTED 
ADMISSIBLE  PROBABILITY  TESTING 

WILLIAM  L.  SIBLEY 
THE  RAND  CORPORATION 
SANTA  MONICA,  CALIFORNIA 


INTRODUCTION 

The  work  reported  in  this  paper  was  conducted  under 
the  Computer  Technologies  in  Training  Project  of  the  ARPA- 
sponsored  Manpower  and  Training  Management  Program  of  the 
Rand  Corporation.  As  indicated  by  the  project  title,  we 
are  interested  in  ways  computer  technology  can  be  used  to 
enhance  the  military  services'  training  program  as  well  as 
to  help  the  selection  and  placement  processes.  In  particu- 
lar, we  have  recently  been  investigating  the  use  of  computers 
in  testing. 

As  reported  to  you  in  1969  by  W.  C.  Gardner,  Jr.  and 
E.  H.  Shuford,  Jr.,  there  is  considerable  experimental 
evidence  that  testing  systems  that  incorporate  admissible 
probability  measurement  have  wide  application  in  important 
areas  of  military  testing  and  evaluation  II,  2].  The  nature 
of  Admissible  Probability  Testing  and  its  extensions  invites 
the  use  of  computers  in  their  application  as  I shall  try  to 
make  clear  in  what  follows. 

The  remainder  of  the  discussion  is  divided  into  three 
parts : 

o A review  of  the  motivations  and  theoretical  founda- 
tions of  Admissible  Probability  Testing  (APT) . 

o A description  of  the  role  of  the  computer  in  APT. 

o A brief  discussion  of  our  experience  to  date  with 
the  computer-based  system. 


MOTIVATIONS  AND  FOUNDATIONS 

The  prime  motivation  for  Admissible  Probability  Testing 
is  simple  and  straightforward:  we  need  to  measure  better 
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the  "state  of  knowledge"  of  trainees  in  a large  variety  of 
subject  areas.  An  improved  measuring  technique  would  be  of 
considerable  value  not  only  in  testing  a trainee  but  also  in 
evaluating  and  planning  curricula.  As  indicated  by  Gardner, 
planning  is  especially  critical  for  those  courses  of  instruc- 
tion that  are  individually  tailored  to  the  trainee  [1] . 

A second,  and  no  less  important,  motivation  is  that  the 
estimation  of  probabilities  is  a critical  skill  in  some  en- 
deavors; for  example,  weather  prediction,  intelligence  analy- 
sis, etc.  I believe  that  it  will  become  apparent  that  APT 
is  especially  applicable  in  the  training  of  students  to  use 
effectively  all  of  their  facts  and  reasons  in  the  estimation 
of  probabilities. 

A corollary  of  the  need  for  training  specialists  in 
probability  estimation  is  the  idea  that,  in  fact,  we  are  all 
engaged  in  probability  estimation.  That  is,  there  is  a need 
for  us  all  to  convey  accurately  and  effectively  our  opinions 
of  the  possibilities  of  various  outcomes.  If  that  is  the 
case,  we  must  learn  to  speak  the  language  of  probabilities 
precisely  and  with  a common  understanding. 

So  what  is  Admissible  Probability  Testing  and  how  is  it 
applied?  I shall  talk  about  APT  in  terms  of  our  specific 
application  even  though  it  has  greater  generality  than  may 
be  apparent  from  my  discussion. 

Consider  the  multiple  choice  test  item  in  Figure  1.  We 
are  generally  required  to  respond  to  such  a question  with 
what  amounts  to  a probability  distribution  as  is  illustrated 
in  Figure  2.  However,  it  would  be  my  personal  tendency  to 
respond  with  the  probability  distribution  illustrated  in  Figure 
3.  In  fact,  if  confronted  with  the  question  in  Figure  4,  my 
response  might  very  well  be,  "I  can't  distinguish  among  the 
alternatives."  That  is,  as  far  as  I can  tell  from  my  facts 
and  reasons,  each  alternative  is  equally  likely.  I think  it 
is  evident  that  requiring  a response  of  zeroes  and  a single  one 
can  mask  a large  variety  of  states  of  knowledge  as  well  as 
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QUESTION;  The  11th  president  of  the  United  States 
was ; 

1.  George  Washington 

2.  Jaunes  Polk 

3.  Franklin  Pierce 

Figure  1.  Multiple  Choice  Test  Item 


QUESTION:  The  11th  president  of  the  United  States 

was ; . 

0 1.  George  Washington 

1 2.  Jcimes  Polk 

0 3.  Franklin  Pierce 

Probability 
Distribution 


Figure  2.  A Standard  Response 


was : 

0 1 1.  George  Washington 

.5  2.  James  Polk 

.5  3.  Franklin  Pierce 

Revised 

Probability 

Distribution 


Figure  3.  A Revised  Response 


QUESTION:  The  11th  president  of  the  United  States 

was : 


.3 

.3 

.3 


1 . John  Tyler 

2.  James  Polk 

3.  Franklin  Pierce 


Figure  4.  An  "unable  to  distinguish"  Response 
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introduce  guessing  on  the  part  of  the  student. 

If  such  freedom  of  response  is  allowed,  there  remains 
the  problem  of  how  to  score  such  a response.  That  is  pre- 
cisely the  function  of  the  "admissible,"  or  "proper,"  or 
"reproducing"  scoring  system  [3] . Such  a scoring  system 
encourages  the  strategy  of  responding  with  exactly  what  you 
perceive  the  probability  distribution  to  be  based  on  your 
own  facts  and  reasons. 

The  admissible  scoring  system  can  be  motivated  in  terms 
of  a gambling  example  (a  more  complete  discussion  can  be 
found  in  [3]).  For  the  sake  of  simplicity,  suppose  that  a 
student  were  faced  with  a true-false  question.  Then  he 
may  be  viewed  as  a gambler  faced  with  determining  the  proba- 
bility of  occurrence  of  the  event  "the  correct  response  is 
’true'."  Suppose  also  that  there  were  a gambling  house  in 
which  there  were  $(u)du  wagers  available  at  odds  (l-u)/u 
(the  "correct  odds"  for  an  event  of  probability  u) . If  the 
student  believes  that  the  probability  of  the  event  taking 
place  were  p,  he  would  take  all  wagers  at  odds  better  than 
(l-p)/p.  Similarly,  he  would  take  all  wagers  on  the  event 
not  taking  place  at  odds  better  than  those  appropriate  for 
probability  1-p. 

Figure  5 illustrates  the  mathematics  of  admissible 
scoring  systems.  Line  (1)  corresponds  to  our  true-or-f alse 
discussion.  Line  (2)  extends  those  notions  to  a n-foil 
multiple  choice  question.  Line  (3)  adjusts  the  equations 
to  give  a score  of  zero  for  the  "unable  to  distinguish" 
response  of  for  i=l,n.  What  these  integrals  repre- 

sent is  the  net  settlement  made  between  the  gambling  house 
and  the  student,  after  the  correct  alternative  has  been  re- 
vealed. It  is  the  sum  of  settlements  on  an  infinite  number 
of  infinitesimal  wagers,  each  of  which  appeared  to  the 
student  to  be  favorable  to  him. 

Figure  6 illustrates  the  logarithmic  admissible  scoring 
system  which  has  the  property  of  depending  on  only  the 
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Figure  5.  Admissible  Scoring  Systems 


probability  ascribed  to  each  foil  independent  of  the 
probabilities  ascribed  to  the  remaining  foils.  Line  (2) 
of  Figure  6 illustrates  the  particular  version  of  the 
logarithmic  system  we  use  in  Admissible  Probability  Test- 
ing. No  probability  less  than  .01  may  be  assigned  to 
avoid  infinitely  large  penalties. 

The  gambling  illustration  shows  that  it  is  in  the 
student's  best  interest  to  gauge  accurately  his  probabili- 
ties and  thus  to  make  no  unnecessary  bets.  That  is,  the 
scoring  system  does  encourage  him  to  reproduce  his  own 
probabilities  accurately. 

However,  how  does  the  student  learn  to  "play  the 
game"  and  improve  the  way  he  brings  his  facts  and  reasons 
to  bear  on  the  determination  of  the  probability  he  ascribes 
to  each  alternative?  As  Gardner  points  out,  "It  seems 
likely  that  the  more  exposure  the  students  have  to  [proba- 
bility] tests,  the  more  realistically  they  will  evaluate 
test  items  and  their  own  knowledge,  thus  providing  a more 
valid  indication  as  to  their  actual  level  of  achievement" 
[1] . We  believe  the  answer  to  the  question  lies  in  the 
computer  system  I will  describe  next. 


THE  ROLE  OF  THE  COMPUTER 

As  you  will  see  in  the  figures  that  follow,  the  com- 
puter-based system  we  have  developed  requires  a computer 
terminal  with  graphics  capabilities  and  some  means  of  allow- 
ing the  student  to  interact  with  it  in  elapsed  times  of  at 
most  a few  seconds.  The  original  system  was  developed  on 
a highly  interactive  console  connected  to  a powerful  IBM 
370/155  computer.  A newer  system  exists  on  a "smart  termi- 
nal," the  IMLAC  PDS-ID,  which  supports  the  system  indepen- 
dently of  any  larger  computer.  My  discussion  will  relate 
to  the  latter  system. 
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Figure  7 illustrates  the  display  the  student  sees 
while  answering  a question.  He  assigns  a probability  to 
each  alternative  by  choosing  a point  in  the  interior  or 
on  the  boundary  of  the  equilateral  triangle.  In  such  a 
triangle,  the  sum  of  the  distances  of  the  point  from  each 
side  of  the  triangle  is  a constant  and  can  be  scaled  to 
1.  Thus  each  point  determines  a probability  distribution 
for  the  three  alternatives.  The  probabilities  appear  at 
the  vertices  of  the  triangle  and  the  appropriate  loga- 
rithmic score  appears  close  by.  Figure  7 shows  the  re- 
sults for  the  "unable  to  distinguish"  case.  The  proba- 
bilities are  constrained  to  sum  to  one.  Also,  the 
probabilities  must  be  greater  than  .01  because  of  the 
behavior  of  the  logarithm  near  zero. 

The  student  may  choose  different  points  in  the  triangle 
until  he  is  satisfied  with  the  indicated  probability  or 
score  distribution.  The  nearer  he  approaches  a vertex, 
the  nearer  that  probability  approaches  .98  and  the  other 
two  approach  .01.  When  he  is  satisfied  with  the  distri- 
bution he  selects  the  NEXT  option  and  the  terminal  displays 
his  cumulative  score  as  illustrated  by  Figure  8. 

The  cumulative  score  is  presented  in  the  form  of  the 
maximum  (or  minimum)  score  attainable  question  by  question. 
In  the  case  illustrated  in  Figure  8,  nothing  has  been  gained 
or  lost  by  the  score  assignments  in  Figure  7 (the  final 
segment  in  the  broken  line  in  Figure  8) . However,  a proba- 
bility of  .98  assigned  to  a correct  answer  would  result  in 
a gain  of  23  points  while  .98  assigned  to  an  incorrect 
alternative  would  result  in  a loss  of  76  points.  Thus,  the 
student  has  question  by  question  feedback  about  the  rela- 
tionship of  his  probability  assignments  to  his  score  and 
may  adjust  his  actions  accordingly.  That  is,  he  may  develop 
a "feel"  for  the  scoring  system.  Under  usual  test  admini- 
stration procedures,  the  delay  from  testing  to  reporting 
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QUESTION: 


The  11th  president  of  the  United  States  was: 


JOHN  TYLER 

You  will  GAIN  0 points 
if  this  is  the  correct  answer. 


.34 


NEXT 


You  will  GAIN  0 points  if 
this  is  the  correct  answer. 

JAMES  POLK 


You  will  GAIN  0 points  if 
this  is  the  correct  answer. 

FRANKLIN  PIERCE 


Figure  7.  Probability  Assignment 
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QUESTION: 

The  11th  president  of  the  United  States  was: 


JOHN  TYLER 

You  will  GAIN  0 points  if 
this  is  the  correct  answer. 


You  will  GAIN  0 points  if  You  will  GAIN  0 points  if 

this  is  the  correct  answer.  this  is  the  correct  answer 

JAMES  POLK  FRANKLIN  PIERCE 


Figure  8.  Cumulative  Score 


the  scores  is  so  long  as  to  prevent  the  development  of  a 
"feel"  for  the  scoring  system.  Moreover,  the  scores  are 
usually  not  reported  on  an  item  by  item  basis.  This 
portion  of  the  computer-based  system  gives  the  student 
a maximum  amount  of  experience  with  the  scoring  system. 

Even  though  the  student  may  learn  to  express  his 
own  probabilities  accurately  the  question  remains  as  to 
how  -^hose  probabilities  relate  to  reality.  That  question 
can  be  answered  in  part  by  his  External  Validity  Graph 
13]  . 

The  External  Validity  Graph  can  be  used  to  deter- 
mine how  well  the  student  does  as  an  estimator  of  proba- 
bility. It  can  be  thought  of  as  his  "track  record"  as 
an  estimator.  We  may  collect  all  those  events  which  he 
claims  to  have  a probability  of,  say,  .80  of  occurring, 
and  determine  what  percentage  of  them  did,  in  fact,  occur. 
If  he  were  a perfect  estimator,  we  would  expect  roughly 
80%  of  them  to  occur.  Figure  9 illustrates  such  a graph. 
The  diagonal  AB  represents  a perfect  estimator.  If  we 
collect,  say  for  20  questions,  all  the  probabilities  used 
by  a student  and  compute  the  relative  frequency  with  which 
each  particular  probability  is  associated  with  a correct 
foil,  we  may  construct  an  external  validity  graph  for  that 
student  and  that  test  material.  We  then  assume  that  the 
relationship  between  them  is  roughly  linear  and  apply  a 
simple  linear  regression.  We  call  the  resulting  line  the 
student's  realism  line.  The  lines  A'B'  and  A"B"  in  Figure 
9 illustrate  possible  results.  The  line  A'B'  has  the 
following  interpretation:  about  50%  of  the  events  given 

a probability  of  1.0  of  occuring  did  in  fact  occur.  We 
interpret  this  to  mean  that  the  student  tends  to  over-value 
his  facts  and  reasons.  Similarly,  the  line  A"B"  can  be 
interpreted  as:  the  events  given  a probability  of  .60  (or 

greater)  of  occuring  did  in  fact  occur  100%  of  the  time. 
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The  student  tends  to  under-value  his  facts  and  reasons. 

We  may  further  interpret  the  realism  line  as:  "when  he 

estimated  x he  should  have  estimated  Ax+B"  where  Ax+B  is 
the  equation  of  his  realism  line.  We  may  then  transform 
the  student's  probabilities  using  his  realism  line  and 
recalculate  his  score  on  the  basis  of  the  new  probability 
assignments.  This  "more  realistic"  score  can  then  be 
used  to  decompose  his  original  score  into  the  portion 
attributable  to  the  mis-estimation  of  probabilities  and 
that  attributable  to  lack  of  information.  The  lower 
portion  of  Figure  10  illustrates  such  a score  decomposi- 
tion. 

The  scores  are  actually  reported  as  the  difference 
between  the  student's  scores  and  the  maximiam  attainable 
for  a given  length  test.  The  over-all  gain  is  that  maxi- 
mum less  the  un-revised  score.  The  gain  attainable  from 
more  information  is  that  maximum  less  the  revised  score. 
The  gain  from  more  "realistic"  use  of  probabilities  is 
simply  the  revised  score  less  the  original  score. 

The  upper  portion  of  Figure  10  brings  us  to  another 
interesting  aspect  of  the  logarithmic  scoring  system. 

We  call  the  square  in  the  upper  part  of  Figure  10  the 
student's  information  square.  The  term  "information" 
arises  from  a very  interesting  relation  between  the  loga- 
rithmic scoring  system  and  the  concept  of  "entropy"  in 
information  theory.  That  relation  is  developed  below. 

Figure  11  illustrates  the  expected  score  for  a given 
question  in  our  three  foil  system.  The  expression  in  the 
parentheses  is  called  in  information  theory  the  "entropy" 
of  the  partition  p^.  It  represents  the  expected  cimount 
of  information  which  will  be  conveyed  by  the  event  itself; 
or,  in  other  words,  the  expected  surprise  content  of  the 
event.  Equation  (2)  of  Figure  11  is  the  expected  score 
normalized  to  the  range  0 to  1.  Equation  (2)  can  be 
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INFORMATION 


YOU  TEND  TO  OVER  VALUE  YOUR  INFORMATION. 

YOU  CAN  IMPROVE  YOUR  SCORE  BY  261  POINTS 
OVERALL . 

YOU  CAN  IMPROVE  YOUR  SCORE  BY  236  POINTS  BY 
GAINING  MORE  INFORMATION  ABOUT  THE  SUBJECT. 

YOU  CAN  IMPROVE  YOUR  SCORE  BY  25  POINTS  BY 
MORE  REALISTIC  USE  OF  YOUR  INFORMATION. 


Figure  10.  Information  Square  and  Score 
Decomposition 


3 

(1)  Expected  Score  = ^ ^ log  3 p^) 

i=l 


3 

= 50  log  3 - (-y^  P^log  p^)  j 
i=l 


3 

(2)  Relative  information  = 50  log  3 - (-^  ^ p^logp^)j 

i=l 

50  log  3 


Figure  11.  Information  Theory  Analogues 


interpreted  as  follows:  if  there  is  no  surprise  content 

in  the  event  then  there  is  perfect  information  Ieq(2)  = 1] ; 
if  there  is  maximum  surprise  content  in  the  event  then 
there  is  no  information  [eq(2)  = 0].  The  upper  part  of 
Figure  10  plots  on  the  PERCEIVED  axis  the  point  corres- 
ponding to  the  relative  information  computed  from  the 
student's  original  probability  distributions.  The  relative 
information  derived  from  the  probabilities  as  transformed 
by  the  realism  line  (the  regression  line  in  the  external 
validity  graph)  is  plotted  on  the  REVISED  axis  and  the 
two  points  are  connected  by  a line.  This  information 
square  has  an  interesting  interpretation  in  terms  of  an 
Arabian  proverb  as  depicted  in  Figure  12. 


EXPERIENCE  TO  DATE 

The  few  observations  I would  like  to  make  here  are 
based  largely  on  anecdotal  evidence.  We  have  used  every 
opportunity  to  test  our  approach  before  committing  to 
formal  experimentation.  We  have  gleaned  our  test  students 
from  among  the  Rand  professional  and  secretarial  staffs 
(and  their  children),  high  school  and  college  students, 
casual  visitors  and  attendees  at  various  Rand-sponsored 
meetings.  It  is  gratifying  to  report  that  not  one  of  our 
subjects  has  had  difficulty  with  the  mechanics  of  using 
the  system. 

Figure  13  illustrates  the  relation  between  the  actual 
and  perceived  information  measures  for  the  first  exposure 
of  66  subjects  to  the  test  system.  This  figure  supports 
what  we  have  observed:  subjects  tend  to  overvalue  their 

facts  and  reasons  at  the  outset.  Our  observation  has  been 
that  most  subjects  tend  to  choose  the  vertices  of  the 
triangle  rather  than  try  to  differentiate  more  finely 
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CHILD 


He  who  knows , and  knows  that  he  knows , 

He  is  wise,  follow  him. 

He  who  knows  and  knows  not  that  he  knows, 

He  is  asleep,  awaken  him. 

He  who  knows  not,  and  knows  not  that  he  knows  not. 
He  is  a fool,  shun  him. 

He  who  knows  not,  and  knows  that  he  knows  not. 

He  is  a child,  teach  him. 


among  their  probability  assignments.  This  seems  to  be  a 
result  of  previous  training  in  answering  multiple  choice 
items . 

Figure  14  illustrates  the  amount  of  improvement  in 
realism  experienced  by  our  subjects.  Among  the  fifteen 
individuals  who  have  taken  a test  on  the  computer  two  or 
more  times,  the  average  loss  in  score  due  to  lack  of 
realism  on  the  first  test  was  108  points;  the  average 
loss  in  score  due  to  lack  of  realism  on  the  last  test 
they  took  was  15  points. 


THE  FUTURE 

The  near  term  future  of  this  work  as  we  see  it  con- 
sists of  at  least  three  parts; 

o An  on-going  improvement  of  our  test  and  inter- 
pretation ideas. 

o Several  small-scale  formal  experiments  to  verify 
our  design. 

o An  implementation  of  our  ideas  on  the  University 
of  Illinois'  PLATO  IV  system. 

This  final  point  indicates  our  desire  to  make  Admis- 
sible Probability  Measurement  available  to  a large  community 
of  potential  users.  Within  that  community  we  hope  to  en- 
courage experimentation  on  a larger  scale  than  we  can  sup- 
port with  our  present  facilities. 


CONCLUDING  REMARKS 

At  the  beginning  of  the  paper,  I made  reference  to  the 
possible  importance  of  Admissible  Probability  Testing  for 
selection  and  training  in  the  military  services.  As  a re- 

if 

suit  of  his  experiment,  Gardner  [1]  found  the  following 
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11  1 1 

Av.  loss  due  to  lack  of  realism  on  first  test:  108 
II  II  II  II  II  It  " " last  " : 15 


hypotheses  to  be  supported; 


o The  scores  attained  on  a [probability]  test  will 
be  a more  accurate  assessment  of  student  knowl- 
edge than  will  choice  scores, 
o [Probability]  test  results  will  be  more  reliable 
than  choice  test  results  since  there  is  significant 
reduction  in  guessing. 

o [Probability]  tests  yield  more  information  that 
can  be  used  in  policy-making  for  a multi-tracked 
curriculum  than  do  choice  tests. 

We  have  taken  these  results  as  an  encouragement  to 
continue  to  develop  and  extend  Computer  Assisted  Admissible 
Probability  Testing. 
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I.  INTRODUCTION 

A.  Purpose 

The  author  realized  that  statistical  theory  and  practice  had  a 
serious  gap;  namely,  that  no  adequate  statistical  technique  existed 
for  the  analysis  problem  associated  with  the  prediction  of  categorical 
membership  from  knowledge  of  other  categorical  (qualitative)  information. 
He  attempted  (Moonan,  1972)  to  provide  an  adequate  solution  to  this 
problem.  His  solution,  augmented  by  a computer  program,  permitted  the 
utilization  of  possibly  several  hundred  available  qualitative 
predictors.  Since  most  operational  situations  employing  prediction 
analysis  cannot  practically  utilize  predictor  information  of  this 
magnitude,  it  was  soon  realized  that  some  model  seeking  technique 
would  have  to  be  developed  in  order  to  make  the  ABCD  technique  more 
effective.  Other  prediction  analysis  procedures  have  their  associative 
model  seeking  procedures  as  illustrated  in  Table  4,  p.  17,  of  Moonan 
(1973) . These  other  variable  selection  techniques  were  not  useful 
for  problems  analyzed  by  the  ABCD  technique  and,  consequently,  a new 
procedure  called  CHAROSEL  was  developed. 

B.  Background 

The  initiai-ion  of  this  development  began  with  a real  personnel 
problem,  namely  that  of  classifying  individuals  into  various 
categories  of  drug  abusers  from  knowledge  of  biographical,  life- 
history,  and  opinion  data  items.  It  may  be  of  interest  to  this 
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audience  to  note  that  both  the  ABCD  and  CHAROSEL  procedures  were 
conceived,  analyzed,  and  programmed  before  data  from  the  project 
were  made  available  for  analysis.  There  are  several  lessons  to  be 
learned  from  this  experience,  none  of  which  can  be  elaborated  here, 
other  than  the  bemusing  consideration  that,  contrary  to  many  research 
analysts'  opinion,  personnel  research  planning  and  data  collection 
activities  are  more  time  consuming  than  data  analysis:  Personnel 

research  planning  and  data  collection  activities  may  indeed  be  much 
more  time  consuming  than  the  development  of  data  analysis  techniques. 

The  type  of  prediction  problem  referred  to  has  been  labeled  by 
the  author  as  a Type  D prediction  problem.  Other  models  for  prediction 
analysis  have  been  identified  by  Lubin  (1950)  and  Rozeboom  (1966)  and 
are  labeled  as  in  Table  1. 


Insert  Table  1 about  here 


1.  What  is  the  kind  of  problem?  Statistical  consultants  and 
analysts  frequently  find  it  useful  to  answer  the  question,  "What 
)cind  of  problem  is  this?"  when  consulting  with  clients  or  developing 
analysis  procedures.  The  procedure  has  several  advantages,  since, 
if  the  question  can  be  answered  succinctly  and  clearly,  the 
complications  associated  with  the  consultation  or  analysis  are 
greatly  simplified.  Accordingly,  we  shall  attempt  to  use  this 
technique  in  our  description.  The  answer  to  the  question  is  most 
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appropriately  the  identification  of  a particular  class  of  statistical 
classification  techniques.  Some  of  the  candidates  are  indicated  as 
follows : 

a.  The  discrimination  problem.  Given  a set  of  distinct 
populations  and  a multivariate  random  variable,  the  discrimination 
problem  is  that  of  defining  a numerical  index  using  the  multivariate 
information  in  such  a way  that  this  index  optimally  discriminates 
among  the  populations.  The  classical  solution  to  this  problem  is 
given  by  the  linear  discriminant  function  of  R.  A.  Fisher  (1936) . 

The  appropriate  model  see)cing  technique  for  this  problem  utilizes 
Mahalanobis'  generalized  distance  statistic  (1936)  as  illustrated 

by  Rao  (1952) . 

b.  The  taxonomy  problem.  If  the  researcher  has  reason  to 
believe  that  his  multivariate  sample  came  from  more  than  one  distinct 
population,  the  taxonomy  problem  is  used  to  evolve  a rule  for  sorting 
the  sample  observations  into  several  distinct  populations.  An  approach 
to  this  problem  solution  for  multivariate  normal  predictors  has  been 
provided  by  Wolfe  (1967)  and  for  simple  binary  attribute  predictors 

by  Tanimoto  (1960) . No  specific  model  seeking  procedures  have  been 
developed  for  these  techniques  or  are  known  to  the  author. 

c.  The  classification  or  diagnosis  problem.  Given  a group 
of  distinct  multivariate  populations  for  which  samples  are  available, 
the  solution  to  the  classification  problem  is  a decision  rule  that 
would  enable  one  to  assign  an  additional  observation  to  his  proper 
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population  if  the  multivariate  observational  vector  is  available  and 
it  is  known  that  the  individual  has  been  derived  from  one  of  the  given 
populations.  There  are  a variety  of  techniques  available  for  appli- 
cation, all  of  which  are  based,  in  one  way  or  another,  on  modern 
decision  theory  and,  for  the  most  part,  utilize  multivariate  normal 
assumptions.  Among  these  solutions  are  the  Wald  (1944)  statistic 
and  the  Anderson  (1951)  approach.  Model  seeking  approaches  within 
the  classification  problem  solutions  are  not  generally  available. 

d.  Types  of  variables . The  concept  of  variables  enter  into 
the  problem  statement  in  two  different  ways.  The  first  way  occurs 
when  consideration  is  given  to  the  fact  that  we  have  both  a criterion 
variable,  i.e.,  category  to  be  properly  identified  for  an  individual 
and  a set  of  predictor  variables  from  which  the  criterion  category 
is  to  be  ascertained.  It  is  also  interesting  to  note  that  our  problem 
is  uniquely  concerned  with  variables  that  are  described  as  "qualitative" 
rather  than  "quantitative."  These  types  of  variables  differ  with  respect 
to  t)ie  way  by  which  their  measurement  is  "manifested."  Qualitative 
variable  measurements  are  manifested  by  a specification  of  a quality 
of  the  variable  on  a nominal  scale  of  measurement.  When  the  variable 
is  manifested  by  or  for  an  object  through  an  item  of  a questionnaire, 
the  response  quality  is  called  an  alternative.  Quantitative  variable 
measurement  is  manifested  on  an  interval  or  ratio  scalable,  metrical 
dimension  as  a quantity.  Our  concern,  in  this  report,  is  exclusively 
with  qualitative  variables  used  both  as  criteria  and  predictors. 
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From  these  considerations  it  definitely  appears  that  the  problem 
at  hand  is  a ciassifioaLion  problem  whose  solution,  because  of  the 
qualitative  nature  of  the  predictor  information,  is  provided  by 
Moonan's  ABCD  technique  for  which,  until  now,  no  model  seeking 
procedure  was  available. 

2.  Problem  statement.  Thus  our  problem,  succinctly  stated, 
is  to  devise  a model  selection  technique  for  prediction  problems 
involving  the  prediction  (or  decision)  of  the  category  of  membership 
using  as  an  information  base  for  prediction  a large  set  of  other 
qualitative  variables. 


II.  THEORY 

A.  Notation 

represents  the  category  of  group  membership,  = 1(1)K. 

(1)  . . . , represents  the  predictor 

qualitative  variables  associated  with  individual  i. 

(2)  P(X^,  6j^)  represents  the  probability  that  individual  ^ 

possesses  predictor  information  X.  and  is  a member  of 

—1 

criterion  category  k. 

(3)  represents  tJie  a priori  probability  of  a randomly 
selected  individual  being  a member  of  category  k. 

(4)  P(x^  I represents  a probability,  known  as  an  endorsement 

ratio,  indicating  the  probability  of  an  individual  from 
category  ^ responding  to  the  alternative  of  the  m*^*^ 
predictor  character. 
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P(X.  , 6j^) 

(5)  Then  P(X.  I 6,  ) = represents  the  conditional 

1 k ^^®k 

probability  of  x.  given  0 . We  are,  however,  primarily 
interested  in  the  "inverse"  of  this  probeibility  which  is 
obtained  from  Bayes’  (1763)  formula  expressed  as  follows: 


P(0^)P«.  I e^) 

(6)  p(e^  I X.)  . . 

I p(x.  )p(x.  1 e,  ) 

k=i  ^ ^ 

With  this  notation  we  are  now  in  a position  to  describe  the 
theoretical  process  of  our  model  selecting  procedure  called  CHAROSEL. 
B.  Procedure  Description 

1 . General . The  CHAROSEL  procedure  computes  a vector  of  the 
Bayes'  formula  probabilities,  P(-j.  I each  member  of  a testing 

sample  for  whom  6^^  is  known.  This  vector  is  vector  multiplied  by  the 

column  elements  of  a cost  of  misclassif ication  matrix  ^ The 

f y 

elements  of  C_  equal  zero  when  6=6'.  Otherwise,  C.  > 0.  This 

f U f V 

vector  multiplication  produces  a set  of  K expected  misclassif ication 

costs,  V^,  and  the  decision  is  that  the  individual  is  a member  of 

category  0 if  V is  the  minimum  of  the  V.  . 

e — e “X 

In  order  to  maintain  computational  feasibility,  it  is  assumed 
that  the  elements  of  X.  are  stochastically  independent  so  that: 


(7)  P(X.  , e.  ) 
1 k 


n p(x  , 0.  ) 

, im  k 
m=l 


2.  Variable  selection  technique.  We  shall  accomplish  the  model 


seeking  procedure  by  characterizing  it  as  a problem  of  selection  of 
the  qualitative  predictor  variables.  A similar  approach  to  this 
class  of  problems  has  been  frequently  used  by  the  author  (Moonan, 

1966,  1967,  1968).  The  optimal  approach  to  this  form  of  solution 
has  long  been  considered  by  the  author  to  be  that  of  dynamic 
programnir.g  similar  to  the  extremely  clever  solution  to  the  clustering 
problem  provided  by  Jensen  (1968) . Unfortunately,  the  dynamic 
programming  solution  to  the  model  seeking  problem  has  eluded  me  and 
I have  had  to  be  content,  as  in  this  case,  with  using  what  I call 
"conditional  sequential  optimization."  This  optimization  procedure 
is  defined  as  follows:  The  program  selects  predictors  in  such  a 

manner  as  to  optimize  a meaningful  objective  function.  The  variable 
is  first  selected  which,  by  itself,  has  the  best  objective  function 
value;  next,  another  variable  is  selected  such  that,  when  combined 
with  the  first  selected  variable,  the  two  produce  a predictor  system 
with  best  conditional  objective  function  value.  Having  found  that 
pair,  a new  third  predictor  is  sought  which,  when  combined  with  the 
previous  selected  pair,  produces  a three  variable  predictor  system 
with  the  best  conditional  objective  function  value.  This  process  is 
repeated  until  all  predictor  variables  are  utilized. 

3.  CHAROSEL ' s objective  function . In  order  to  discuss  the 
objective  function  of  CHAROSEL,  it  is  necessary  to  consider  Table  2, 
which  portrays  the  results  from  applying  the  ABCD  technique  for  any 
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Insert  Table  2 about  here 


given  number  of  predictors  in  a three  criterion  category  predict  i 
problem.  Table  2 may  be  interpreted  as  follows:  There  are  three 

criterion  categories  labeled  1,  2,  and  3,  for  which  testing  sample 
sizes  of  39,  24,  and  72,  respectively,  are  available.  Using  ABCD 
for  some  specified  set  of  predictor  variables,  30  of  category  1, 

17  of  category  2,  and  61  of  category  3 were  correctly  classified. 

If  the  predictor  variables  were  completely  effective,  the 
principle  main  - diagonal  (30,  17,  61)  would  be  represented  by 
(39,  24,  72),  showing  that  all  testing  sample  members  were  correctly 
classified  and  all  non-diagonal  elements  would  have  frequency  = 0. 

This  suggests  that  the  proper  objective  function  is  a 
statistic  representing  this  condition.  One  such  statistic  is 
provided  by  an  application  of  classical  information  theory  and  was 
suggested  by  the  author  to  his  programmer,  Mr.  Samuel  Bowser.  This 
statistic  proved  inappropriate  for  some  applications  and  was  modified 
by  Mr.  Bowser  to  equation  (8) . 

(8)  Objective  Function 


Z I Pi  Z P, .Log  (P. .)  I + ^ ii 


i I PI  r p . . rog.,  (P . . 

i=l  j=l  2 13 

[ 17^3 


i=l  f . . 
11 


Lim  1 . 
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Lim  f , . 
11 


n . , 

11 


n.  . 
11 

f . . 

11 


= A priori  probability. 

= Cell  proportion  off  diagonal 
n^^  = Number  of  occurrences  in  a given  group 
f^^  = Diagonal  cell  frequency 
k = Number  of  groups 
Lim  X log  x = 0 
X ^ 0 


Equation  (8)  has  a value  = 1.00000  if  the  decision  tables' 
diagonal  equals  the  frequencies  of  the  testing  samples'  category 
sizes,  otherwise  the  statistic  is  greater  than  unity. 


III.  IMPLEMENTATION 

A.  The  CKAROSEL  Technique : Computer  Program  Documentation 

The  CHAROSEL  technique  was  programmed  as  a computer  program  for 
the  NELC  IBM  360/65  system  and  can  be  executed  through  the  NPRDC 
IBM  1130  remote  job  entry  system.  The  documentation  for  this  program 
follows  as  extracted  from  Bowser  (1973)  and  Moonan  and  Bowser  (1973) . 

B.  Identification 

Title:  CHAROSEL 

Common  Identification:  CHARI 

Category:  Statistical 

Programmer:  Samuel  E.  Bowser 
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Analyst:  William  J.  Moonan 

Date:  1 May  1973 

Compiler:  IBM  360/65  FORTRAN  G 

Core  Requirements:  375K 

C.  Usage 

1 . Input  deck  setup 

a.  Program  deck  and  subroutine  decks. 

b.  Input  form  including  parameter  and  option  cards. 

c.  Data  either  card  or  9-track  tape. 

2.  Input  card  layout 

See  input  forms,  Bowser  (1973) . 

3.  Equipment  configuration 

Input  form  cards  are  always  loaded  behind  the  program  deck 
on  the  standard  input  unit.  Input  data  can  be  on  a separate  tape  or 
loaded  from  the  standard  input  unit. 

4 . Output 

a.  Printer 

(1)  Decision  table  as  each  variable  is  selected. 

(2)  Values  of  objective  function,  change  in  objective 
function,  total  per  cent  correct  decisions,  and 
change  in  percentages  of  correct  decisions. 

(3)  Variables  are  listed  in  rank  order  as  they  are 
selected. 
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b.  Punch 


(1)  Endorsement  ratios  are  punched  out  if  they  are 
calculated  by  program;  this  option  is  selected 
unless  otherwise  specified. 


D.  Requirements 

1.  Storage.  The  program  as  currently  constructed  does  not 
utilize  object  time  dimensioning.  The  size  of  the  core  requirement 
at  present  is  375K.  This  allows  for  500  men  and  200  variables.  The 
number  of  categories  is  fixed  at  10  or  less  and  the  levels  of  the 
variables  is  fixed  at  10  or  less.  In  order  to  exceed  the  size 
restrictions  the  redimensioning  of  the  main  program  and  its  subroutines 
would  be  necessary. 

2.  Time.-  An  approximation  of  the  time  required  can  be  obtained 
from  the  following: 


Time  (Min.)  = 5 


(Nr.  of  Variables)' 
300 


14  ) (Nr.  of 


Categories  - 1) 

Thus,  for  a three  category,  50  predictor  variable  problem, 
the  computing  time  of  CHAROSEL  for  normal  options  is  about  17  minutes. 

For  many  other  imp'ortant  details  regarding  documentation  and 
instructions  for  filling  out  the  input  forms  the  user  should  consult 
Bowser  (1973) . 

E.  Program  Restrictions 

1.  The  training  sample  size  must  be  less  than  or  equal  to  500. 
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2.  The  number  of  criterion  categories  must  be  less  than  or  equal 


to  10. 

3.  The  number  of  levels  for  each  predictor  character  must  be 
less  than  or  equal  to  10. 

4.  The  number  of  predictors  is  limited  by  computer  system  core 
size  and  is  related  to  sample  size.  Adjustment  may  be  made  to  larger 
numbers  of  predictors  but  at  present  the  program  is  set  for  200 
predictor  variables  maximum. 

5.  At  present  the  program  is  operational  only  on  an  IBM  360/65 
computer  system  and  utilizes  a core  size  of  300K  bytes. 

6.  Prospective  DOD  users  may  arrange  for  program  execution 
through : 

Head,  Computer  Services  Department 

Navy  Personnel  Research  and  Development  Center 

San  Diego,  California  92152 

7.  Other  potential  users  should  contact  the  authors. 

F.  User  Options 

The  user  of  CHARCSEL  has  considerable  freedom  with  regard  to  type 
of  procedure  he  wishes  to  utilize  with  this  program.  Among  these  options 
are : 

1.  The  endorsement  ratios  may  be  either  supplied  by  the  user  or 
the  program  will  calculate  them  from  the  training  sample. 

2.  A limit  may  be  given  for  the  maximum  number  of  variables  with 
missing  data  that  are  allowed  for  each  object  of  the  training  sample 

495 


I 


w 


and/or  missing  data  may  be  used  as  a "level"  in  the  corresponding 
predictor  variable. 

3.  Certain  predictor  variables  may  be  forced  to  be  utilized 
initially  in  the  CHAROSEL  predictor  variable  identification  at  the 
option  of  the  user. 

4.  If  the  user  wishes  to  utilize  the  cross  validation  process 
this  may  be  accomplished  by  taking  the  variables  selected  by  CHAROSEL 
and  running  them  on  the  ABCD  program  with  a testing  sample. 

5.  Training  sample  data  may  be  either  on  cards  or  on  tape, 
however,  there  exists  an  input  form  to  be  completed  by  the  user 
before  execution. 

G.  ^ Illustration 

1.  The  problem  description.  As  an  illustration  of  the  CHAROSEL 
program,  we  present  a problem,  actually  executed  on  the  program  and 
provide  the  output  results  obtained  thereby.  This  illustration  is 
concerned  with  the  educational  problem  of  identifying  intellectually 
gifted  children  who  are  members  of  so-called  low  socio-economic 
status  groups  within  the  city  of  San  Diego.  This  illustration  is 
the  PhD  thesis  problem  of  Mrs.  Charlotte  Malone  at  the  United  States 
International  University,  San  Diego.  As  yet,  the  thesis  is  incomplete 
and,  of  course,  unpublished,  and  the  reference  to  this  work  is  only 
provided  for  illustrative  purposes.  The  categories  of  criterion  for 
this  problem  are  two:  6^^,  the  children  who  have  Stanford  Binet  IQ's 

greater  than  or  equal  to  132  and  labeled  "gifted"  and,  those 
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children  who  have  Stanford  Binet  IQ's  less  than  132  who  are  labeled 
"non-gifted . " 

The  predictor  information  was  derived  from  a questionnaire 
completed  by  the  mothers  of  kindergarten  children  registering  for 
school  within  a certain  school  district  of  San  Diego  during  September 
1973.  The  items  of  the  questionnaire  were  composed  of  51  biographical, 
behavioral,  and  life  history  items  such  as: 

(1)  Does  your  child  use  a lot  of  "grown-up"  words?  Does 

he  talk  like  an  older  person  than  he  really  is? 

A.  Seldom 

B.  Occasionally 

C.  Often 

D.  Almost  always 

E.  No  answer  possible 

2.  The  results.  The  training  sample  for  this  problem  is  at 
present  very  small  but  will  be  augmented  within  a short  time.  For 
the  present,  N^  = 28  gifted  students  and  N^  = 21  non-gifted  students 
on  whom  data  were  available.  The  researcher  specified  the  a priori 
probabilities  to  be  £(S^)  = -05  and  £.(02^  “ .95.  Each  item  of  the 
questionnaire  had  five  levels  of  response  and  their  endorsement  ratios, 
for  each  category,  w'ere  computed  by  CHAROSEL.  The  execution  of  the 
program  produced  the  following  results:  (See  Table  3). 


Insert  Table  3 about  here 


We  note  that  the  decision  table  showed  17  misclassif ications  of  gifted 


children  and  no  misclassif ications  of  non-gifted  children  and  that  the 
most  optimal  single  predictor  item  was  item  number  1,  shown  previously. 

The  objective  function  value  was  1.78957  and  per  cent  correct  accuracy 
= (32/49)100  = 65.31. 

The  program  proceeded  to  sequentially  select  items  29,  7,  and  j 

27,  producing  the  information  outlined  in  Table  4.  , 


Insert  Table  4 about  here 


At  tills  stage,  only  eight  misclassifications  were  made  and  the 

objective  function  has  been  reduced  to  1.21067  and  the  per  cent 

41 

correct  accuracy  = (^^)100  = 83.67. 

After  30  stages  of  selection  the  main  - diagonal  of  the 
decision  table  was  (28,  21),  indicating  that  the  per  cent  correct 
accuracy  of  classification  was  100.00.  The  statistical  results  for 
this  stage  of  analysis  are  available  but  not  shown  in  this  report. 

The  author  has  experienced  the  greatest  personal  satisfaction 
as  a result  of  the  utilization  of  his  CKAROSEL  technique  to  this 
identification  problem  wl  _ch,  if  these  preliminary  results  are 
sustained,  will  lead  to  better  educational  practices  and  improved 
well-being  for  many  children. 
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The  CHAROSEL  program  has  been  applied  to  several  military 
and  civilian  personnel  research  problems.  Some  of  these  results  are 
shown  in  Table  5. 


Insert  Table  5 about  here 


IV.  CONCLUSION 

The  CHAROSEL  technique  represents  a unique  development  in 
classification  theory  in  that  it  possibly  provides  the  only  practical 
procedure  available  for  selecting  effective  qualitative  predictor 
variables  for  Type  D prediction  problems.  A computer  program, 
associated  with  this  technique,  has  been  developed  and  applied  to 
several  problems  associated  with  both  military  and  civilian  personnel 
research.  The  results  from  these  applications  have  shown  the  technique 
can  identify  a few  predictor  variables  which  will  produce  results  of 
very  high  classification  efficiency.  This  report  provided  information 
relative  to  the  theory,  implem.entation , and  application  of  the  program. 
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ADDENDUM 

After  this  report  was  completed,  the  following  thoughts  occurred 
to  the  author . 

The  controlling  aspect  of  CHAROSEL  is  its  objective  function. 

Suppose  this  was  modified  from  equation  (8)  to  the  bivariate  uncertainty 
measure 

k k 

U(.,.)  =-15;  p(i,j)log  p(i,j)  . 
i=l  j=l 

where  p(i,j)  is  the  joint  probability  of  row  i and  column  j in  the 
CHAROSEL  decision  table.  The  objective  of  the  procedure  is,  then,  to 
select  variables  that  maKimize  joint  uncertainty,  U(.,.).  Used  in  this 
way  the  progiam  could  be  used  to  select  variables  to  affect  heterogeneity 
among  categories  in  the  sense  that  the  program  will  identify  those 
variables  which  tend  to  make  an  independent  relationship  between 
predicted  category  and  actual  category. 

This  approach  appears  to  have  application  to  some  analysis  problems 
encountered  at  NPRDC. 
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TABLE  2 

Illustrative  Decision  Table  for  a Three  Category  Problem 


True 

Categories 

2 

3 

Total 

Percent 

Category  1 
Frequencies 

30 

4 

5 

39 

28.9 

Category  2 
Frequencies 

4 

17 

3 

24 

17.8 

Category  3 
Frequencies 

6 

5 

61 

72 

53.3 

Column  Totals 

40 

26 

69 

135 

Column  Percents 

29.6 

19.3 

51.1 

100 

TABLE  3 


CHAROSEL  Decision  Table  for  First  Stage  Analysis 
For  the  Gifted  Children  Problem 


Stage  Variable  Objective  Function  Percent  Correct 

Selected  Name  Nr.  Value  Change  Amount  Change 


TABLE  4 


CHAROSEL  Decision  Table  for  Fourth  Stage  Analysis  for 
The  Gifted  Children  Problem 


True 

Decision 

Category 

Categories 

1 

2 

Total 

Percent 

Category  1 
Frequencies 

20 

8 

28 

57 

Category  2 
Frequencies 

0 

21 

21 

42 

Column  Totals 

20 

29 

49 

Column  Percents 

40 

59 

Stage 

Selected 

Variable 

Name 

Nr. 

Objective 

Value 

Function 

Change 

Percent 

Amount 

Correct 

Change 

1 

01 

1 

1.78597 

0.0 

65.31 

0.0 

2 

29 

29 

1.38743 

0.39855 

75.51 

10.20 

3 

7 

7 

1.24807 

0.13936 

81.63 

6.12 

4 

27 

27 

1.21067 

0.03740 

83.67 

2.04 
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A FACTOR  ArJALYTIC  APPROACH  TO  THE  CRITERIOTJ  PROBLEM 


I . I ntroduct i on 

The  problem  of  how  to  deal  with  multiple  criteria  when  attempting  test, 
or  test  battery,  validation  is  a topic  of  continuing  concern  (e.g.  Blum 
and  Haylor,  1968;  James,  1973;  t^unnaliy,  1967;  Roseboom,  1966;  Schmidt  and 
Kaplan,  1971).  In  the  most  complex  case,  the  predictor  set  (PS)  and  the 
criterion  set  (CS)  contain  multiple  measures.  The  development  of  an  optimal 
set  of  weights  to  obtain  a maximal  correlation  between  the  two  sets  is  well 
known  from  the  theory  of  canonical  correlation  (e.g.  Morrison,  1967). 

This  solution,  however,  answers  the  wrong  question.  In  obtaining  measurement 
on  a CS,  one  attempts  to  utilize  variables  which  measure  a construct, 
namely,  "goodness"  of  job  performance,  say.  Thus,  the  intent  is  to  find  a 
weighting  of  the  CS  which  maximizes  the  congruence  of  the  criterion 
composite  (CO  with  this  hypothetical  construct.  One  begs  the  question 
if  he  employs  a canonical  correlation  approach,  since  consideration  of 
the  job  performance  construct  is  lacking. 

A paper  by  James  (1973)  takes  construct  validity  as  a point  of 
departure  in  discussing  the  criterion  problem.  He  considers  four  models: 
the  so-called  ultimate  criterion  model  which  combines  all  available  criterion 
measures  into  a single  composite,  the  multiple  criterion  model  which  seeks 
to  better  understand  the  interactions  between  the  PS  and  CS  by  maintaining 
the  individuality  of  the  CS  measures,  the  general  criterion  model  which 
merely  formalizes  the  notion  that  the  criterion  construct  is  likely  to  be 
complex  and  multidimensional,  and  fhe  construct  validation  model  which 
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e'^phasijei  the  c.tif-jrion  pfODlen  should  really  be  couched  in  a 

constri..t  validation  rramework.  James  rejects  the  ultimate  criterion 
model  as  be : nq  "erroneous",  perhaps  making  the  tacit  assumption  that  any 
sucn  crit--.-ion  cannot  have  the  requsite  construct  validity.  Furthermore, 
he  suggests  that  "advanced"  methods  of  analysis,  such  as  discriminate 
analysis,  obviate  the  necessity  of  ultimate  criterion  development  and,  thus, 
one  is  nciturally  led  to  a multiple/construct  valid  kind  of  approach.  It 
is  completely  unclear  to  this  author  how  discriminate  anaivsis  is  a useful 
tool  foi'  eiUier  construct  validation  or  PS  validation.  It  is  precisely  this 
kind  of  superficial  analysis  w.h  i ch  provides  the  fuel  necessary  to  sustain 
a potentially  reconcilable  controversy.  Construct  validity,  indeed,  is  a 
totally  separable  issue  apart  from  the  ultimate/multiple  criterion  issue. 

The  fact  is  tfrat  construct  validity  is  a premise,  not  an  arguable  point 
(how  to  achieve  it  is  another  matter),  dith  this  premise  in  mind,  a 
reasonable  anj  desirable  goal  of  a pracf i f i oner  would  be  to  explore  CS 
predi  c *■  t:  i ! ; ‘ : ; or  t.j  i ra  I y , 1 i s cr  i-r  i tr  a te  analysis  is  not  the  answer  unless 

there  is  b;f  r sinqle  criterion  measure  which  can  be  di cotom i zed  on  a 
success-td  i I ir- e dimensior.  Fur  ther-more,  it  would  seem  that  ultimate 
criterion  advocates  are  regarded  as  carefully  avoidinn  the  CS  data  until 
an  issis)-)^  ta-  combined  it  into  the  required  form,  thus  obscuring  or 
e I im  irrat  i n<)  potentially  interesting  relationships.  Certainly,  the 
pract  i t i orrer  will  explore  “^he  re  I at  ionsh  ips  existing  in  his  data  to  as 
detailed  a level  a-  his  still  or  computing  capability  permits,  whether  or 
not  he  finally  elects  to  ♦om  e single  CC. 


This  paper  assumes  the  construction  of  a so-called  ultimate 
criterion  is  the  goal,  and  will  provide  a method  of  accomplishing  this 
which  is,  in  some  sense,  optimal.  The  assumptions  concerning  construct 
validity  will  be  delineated  by  way  of  a two  dimensional  framework  for 
the  CS  which  provides  natural  implications  for  the  appropr i ateness  or 
i nappropri ateness  of  various  prediction  considerations. 

2.  Conceptual  Framework  for  Criterion  Variables 
The  first  dimension  of  interest  for  conceptualizing  the  CS  concerns 
fhe  number  of  measures  in  the  set.  In  the  simplest  case  the  CS  is  but 
a single  measure;  subsequent  analysis,  obviously,  will  depend  solely  on 
the  adequacy  of  this  measure.  Validation  efforts  using  the  PS  utilize 
either  bivariate  linear  regression,  if  N(PS)=  I,*  or  multiple  linear 
regression,  i fi(PS)>l.  This  situation  is  obviously  the  most  desirable 
^rom  d computational  point  of  view,  however,  the  results  may  be  totally 
i rre  levan  + , \ the  criterion  measure  lacks  construct  validity.  The 
other  end  of  this  dimension  is  defined  by  the  case  fi(CS)>l.  Here  the 
classical  problem  of  weigiiting  the  CS  to  obtain  the  "best"  criterion 
measure  is  encountered. 


weighting  are  avoided.  When  rvl(CS)>l,  one  of  three  situations  can  obtain: 
Case  I,  the  CS  can  be  regarded  as  an  exhaustive  set  of  measures 
of  the  construct.  Here  one  must  judge  that  no  additional  measure 
could  add  predicitive  validity  relative  to  the  construct.  Case  II, 
the  CS  may  be  regarded  as  a random  sample  of  measures  potentially 
related  to  the  construct.  This  is  a situation  similar  to  test 
construction  where  an  attempt  is  made  to  include  items  which  constitute 
a more  or  less  random  sample  from  the  domain  of  interest.  Case  III, 
the  CS  may  be  regarded  as  an  at-hand  or  non-random  sample  of  measures 
potentially  related  fo  the  construct. 

this  two  dimensional  structure  for  the  CS  makes  it  clear  where 
potential  solutions  for  criteria  weighting  are  possible  and  useful. 

When  .'Uo'  . !,  -he  only  concern  surrounds  construct  validity.  When 

N(CS)>I,  s,  lutions  are  possible  only  for  Cases  I and  II.  For 
Case  III,  no  wei  ihting,  optimal  or  non-optimal  can  produce  a CC  suitable 
for  f'S  3 ' i dat  i .an  . 

ihe  followin.)  section  will  brie‘ly  outline  the  nature  of  solutions 
which  have  been  p-oposed  for  'i(CS)>l,  mostly  without  regard  to 
whether  Casia  I,  I I or  M I is  appropriate. 

5.  Historical  Solutions  for  !J(CS)>I 

The  f i '■St  class  of  solutions  will  be  called  Simple  Weighting 
Solutions.  The  le.ast  elegant  member  of  this  class  is  the  CC  formed 
by  usini  eqi;al  weights  for  the  c;S  set.  This  solution  attempts  to  optimize 
no  thiiig,  r, either  validity  A/itfi  the  PS  nor  construct  validity. 

A commor.  approa.h,  simil-it  to  *-he  one  above,  is  to  use  nominal  weiahts 
developed  ; / .uii  io.,t  I vo  I evaluating  the  probable  import-ince  of  each 
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CS  measure  relative  to  the  construct  of  interest.  This  solution  attempts 
to  maximize  the  extent  to  which  the  CC  measures  the  construct,  thus, 
optimizing  construct  validity.  Another  member  of  the  simple  weighting 
class  uses  the  reliability  of  each  CS  member  as  its  weight.  This  method 
takes  advantage  of  the  fact  that  reliability  is  directly  a function  of 
true  score  variance,  i.e.  the  higher  the  reliability,  the  higher  the 
relative  true  score  variance.  Thus,  such  a weighting  scheme  maximizes  the 
proportion  of  true  variance,  under  the  constraint  that  all  the  CS  measures 
are  included  in  the  CC. 

All  of  the  simple  weighting  methods  suffer  at  a minimum  from  a lack  of 
consideration  of  the  matrix  of  correlations  among  the  CS.  If  the 
CS  exhibits  any  appreciable  cross-corre I at i ons  (as  they  in  general  will  if' 
construct  validity  has  any  tenability),  then  any  simple  weighting  of  the 
individual  variables  will  produce  a CC  of  unknown  composition. 

Often,  any  sys+em  of  weighting  which  permits  a different 
con f i gu rat i on  of  scores  to  achieve  the  same  CC  score  is  not  justifiable. 

An  example  would  be  a sifuation  in  which  a minimum  performance 
on  ea-h  member  of  the  CS  must  be  obtained  before  performance  is 
considered  acceptable.  This  would  be  the  case  for  pilot  performance, 
for  insfance,  where  minimum  levels  of  accomplishment  for  all 
performance  components  must  be  attained.  The  method  commonly  employed 
is  the  multiple  cut-off  method.  Each  component  of  the  CS  is 
usually  judged  to  have  obvious  content  and  construct  validity. 

Other  approaches,  here  called  empirical  weighting,  use 
weights  which  maximize  a mathematical  criterion.  An  example. 
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previously  alluded  to,  is  the  metliud  of  canonical  correlation. 

This  iTietfiod  suffers  the  fundamental  flaw  of  ignoring  the  construct 
while  maximizing  predictability.  An  empirical  weighting  scheme  developed 
by  Kurtz  (1937),  for  the  Case  N(PS)>1  and  N(CS)>1,  develops  a set  of 
weights  for  the  PS  such  that  the  so  weighted  composite  PS  is 
maximally  related  to  each  member  of  the  CS.  This  method  seems  to 
answer  a question  one  would  tend  not  to  ask,  since  we  discover 
neither  the  maximal  relation  of  predictors  so  weighted  to  any  of  the 
criterian  measur'es,  nor  which  of  the  PS  are  actually  functioning 
as  predictors.  A related  method,  which  has  considerable  theoretical 
justification  in  practice,  is  the  utilization  of  weights  which  are 
proportional  to  the  first  principal  component  of  the  CS.  The 
first  principal  component  of  a test  battery  is  produced  by  finding  a 
set  of  v/eights  (with  unit  variance)  such  that  when  applied  to  the 
origional  measiires,  we  produce  a new  variate  having  maximal  variance. 

For  Case  1 and  II,  this  approach  defines  a theoretically  defensible 
method  tor  constructing  a CC,  providing  the  criterion  domain  is 
unidiiiieiioioiiai . The  latter  consideration  is  no  doubt  rarely  the 
case  in  practice,  i.e.  most  would  agree  tliat  job  success,  say, 
represents  a multidimensional  syste=^  such  that  many  facets  contribute 
to  success  (cf . , Ghiselli,  1964).  These  considerations  lead  us  to 
the  question  of  tiie  factorial  complexity,  in  a factor  analytic 
sense,  of  tiie  criterion,  and  to  a general  solution  of  the  criterion 
v/eighting  problem. 

4.  Factor  Analytic  Weights  for  Multiple  Criteria 
The  theory  of  psychometric  inference  will  be  helpful  to  our 
development  of  a general  weighting  scheme.  Psychometric  inference  is 
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more  readily  understood  when  contrasted  with,  classic  statistical 
inference.  Statistical  Inference  attempts  to  make  judgements  about 
the  value  of  a parameter,  or  set  of  parameters,  in  a population  of 
potentially  measurable  objects  (people,  light  bulbs,  plots  of  land, 
etc).  These  inferences  are  made  by  selecting  a random  sample  of  these 
objects,  and  using  the  value  of  the  parameter  in  the  sampl e to  make 
inferences  about  the  parameter  in  the  population.  Psychometric 
inference,  on  the  other  hand,  seeks  to  make  inferences  about  a 
population  of  tests  or  measures  based  on  a sample  of  tests  at  hand 
(cf . , Kaiser  and  Caffrey,  1965).  Specifically,  we  wish  to  know  if 
results  from  a factor  analytic  study,  i.e.  a set  of  factors,  has 
general i zabi 1 i ty  in  the  sense  that  another  sample  of  tests  from  the 
same  domain  or  nopulation  would  tend  to  render  the  same  factors.  Thus, 
we  make  inferences  about  population  factors  from  a sample  of  tests 
defining  the  factors  in  the  population. 

We  can  be  quite  sure  about  the  general f zabi li ty  of  our  results 
if  we  sample  exhaustively  from  the  population,  or,  obtain  a 
suff i cietitiy  large  random  sample  of  measures.  These  are  exactly 
the  conditions  required  for  Case  I and  II  construct  validity.  The 
question  can  then  be  asked,  suppose  ue  obtain  a factor  analysis  of 
the  CS,  assuming  Case  I or  II  holds.  Is  there  a theoretically 
justifible  CC  which  can  be  constructed  from  this  factor  analysis? 

The  answer  is  yes,  and  the  CC  is  constructed  as  follows  (for 
equational  development  see  Appendix  I). 

Factor  analyze  the  CS  (some  methods  of  accomplishing  this  are 
preferable  to  others);  rotate  the  obtained  solution  to  a nsycitologically 
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pleasing  orientation  (this  can  be  accomplished  using  one  or  another 
analytic  procedures  such  as  Varimax;  Kaiser,  1958).  The  next 
step  is  ^0  compute  the  associated  factor  scores  based  on  the  rotated 
factors.  These  scores  represent  scores  for  people,  on  measures 
(factors)  derived  from  linear  combinations  of  the  basic,  observed 
test  scores.  Now  we  estimate  factor  saliencies  by  computing  the  sum 
of  squared  factor  loading  for  each  factor.  These  saliencies  will 
represent  a set  of  weights  for  the  factor  scores.  Let  the  CC  be 
defined  as  the  sum  of  the  weighted  factor  scores. 

The  me  el  we  are  proposing  says  that  when  case  I or  II  can  be 
assumed  to  hold,  the  results  of  a factor  analysis  of  the  CS  can 
be  said  to  estimate  the  test  domain  factors.  Given  this,  the 
weighting  problem  is  solved  as  a natural  byproduct  of  the  analysis, 
namely,  tiie  weights  are  defined  by  the  estimated  importance  of  the 
factors  in  the  domain.  It  should  be  noted  that  these  weights  are 
relative,  and  not  invariant  or  absolute.  The  magnitude  of  the  weights 
will  be  governed  by  the  initial  scaling  imposed  on  the  covariance  matrix 
of  the  CS.  If  itie  CS  correlation  matrix  is  factor  analyzed,  the  weights 
will  be  different  in  magnitude  from  the  factor  weights  from  the  covariance 
solution,  say.  However,  one  typically  does  not  ask  for  absolute 
invariance,  but  rather,  invariance  up  to  a multiplicative  scaling. 

Note  that  the  present  model  includes  as  a special  case  the  first 
principle  coi;inonent  \;eighting  scheme.  When  the  criterion  domain  is  found 
to  be  uni dimensi onal , or  to  contain  one  common  factor,  weighting  the 
tests  according  to  the  elements  of  this  component  is  identical  to 
producing  a sot  of  factor  scores  based  on  one  common  factor. 
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It  should  be  noted,  as  well,  that  a common  factor  analysis  is 


implyed  by  the  present  model.  Common  factor  analysis  attempts  to 
analyze  variance  common  to  two  or  more  variables.  This  is  accomplished 
by  estimating  the  unique  variance  contained  in  each  measure  in  the  CS , 
and  partialing  this  variance  out  before  the  analysis  is  begun,  (cf. , 

Harman,  1967;  Muliak,  1972.) 

5.  Discussion 

At  this  point  it  seems  prudent  to  clear  up  a confusion  v/hich 
occasionally  occurs  in  the  literature  (cf.  Schmidt  and  Kaplan,  1971). 

The  confusion  surrounds  the  distinction  between  factorially  pure 
measures  and  orthogonal  or  uncorrelated  measures.  The  Schmidt  and 
Kaplan  article  attempts  to  argue  along  the  lines  that  it  is  generally 
conceded  that  simply  weighted  composites  result  in  lower  validity  unless  the 
measures  are  factorially  pure.  But,  they  continue,  the  validity  of  a test 
(defined  as  the  sum  of  the  cross  products  of  the  approoriate  factor 
loadings)  can  under  various  circumstances  be  higher  for  factorially 
complex  tests.  Therefore,  the  composite  will  not  in  general  have  lower 
validity. 

The  confusion  evident  in  this  reasoning  is  difficult  to  sort  out 
entirely,  but  seems  to  be  based  on  the  distinction  between  factor  pure 
measures  and  uncorrelated  measures.  A measure  can  be  factor  pure  and 
still  be  correlated  with  anotlier  variable,  similarily  factor  pure, 
because  they  load  on  the  same  factor.  In  any  event,  it  is  safe  to  say  that 
a composite  variable,  formed  from  correlated  or  uncorrelated  component 
variables,  using  any  simple  weighting  rule,  will  have  decreased  validity 
from  the  theoretically  maximum.  Tiie  comoosite  having  maximal  validity 
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with  the  PS  is  defined  by  the  weights  estimated  via  canonical  correlation, 
if  H(PS)>1,  or  the  multiple  regression  of  the  CS  on  the  predictor,  if 
N(PS)  = 1.  This  is  plainly  obvious  when  N(PS)  = 1;  any  set  of  weights 
departing  from  regression  weights  for  the  CS  produces  lov/er  validity, 
since  these  weights  maximize  the  obtained  correlation.  However,  these 
coi.ci derations  are  largely  misplaced  since  one  is  not  interested  in 
optimally  weighting  the  CS  to  produce  maximal  validity  with  the  PS. 

What  we  are  attempting  is  to  optimally  v/eight  the  CS  so  as  to  be  maximally 
congruent  with  the  relevant  hypothetical  construct.  Tt  seems  to  be  a 
case  of  misplaced  validity. 

Related  to  this  matter  is  the  issue  of  the  so  called  criterion 
oriented  approach  to  test  construction  (Nunnally,  1967).  Here,  one 
attempts  to  construct  items  for  use  in  a predictor  test  which  are 
maximally  correlated  with  a known  criterion.  Such  a set  of  items  can 
then  be  expected  to  produce  maximal  validity  between  a predictor, 
constituting  these  items,  and  the  criterion.  Nunnally  (1967)  argues 
that  this  criterion  oriented  predictor  is  likely  to  be  very  complex 
factorially,  and  lead  to  little  conceptual  understanding  about  the 
attributes  which  are  important  for  criterion  prediction.  He  argues  that 
off-the-shelf,  pure  measures  are  more  likely  to  increase  our  understanding 
of  the  entire  process , as  well  as  producing  comparable  validity.  This 
kind  of  thinking  apparently  prompts  Schmidt  and  Kaolan  (1971)  to 
recommend  that  increasing  the  factorial  purity  of  both  the  PS  and  CS 
will  tend  to  resolve  the  criterion  problem  in  general. 
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This  latter  argument  makes  the  tacit  assumption  that  decreased 


factorial  complexity  on  both  sides  of  the  equation  will  result  in 
decreased  complexity  in  prediction.  Indeed,  the  independent  construction 
of  pure  measures  on  both  sides,  will,  in  general,  not  result  in  simpler 
predictive  relations  unless  the  factors  on  one  side  happen  to  be 
isomorphic  to  those  on  the  other.  This  may  occur  in  practice,  but 
surely  constitutes  an  assumotion  to  be  verified.  Furthermore,  striving 
for  factorially  pure  criterion  measures  may  be  a laudable  goal,  but 
it  is  often  the  fact  that  criteria  constitute  givens.  Just  satisfying 
the  conditions  for  Case  I and  II  may  tax  the  imagination  of  an  innovative 
test  designer  without  striving  for  the  nicety  of  factorial  purity  as  well. 

Achieving  clarity  in  the  multivariate  relationship  between  the 
PS  and  the  CS  constitutes  the  primary  motive  underlying  theoritical 
discussions  about  the  factorial  composition  of  the  two  data  sets. 
Considerable  insight  along  these  lines  can  be  attained  using  the  method 
of  factor  extension  ([\jyer,  1937).  Once  a satisfactory  factor  solution 
has  been  obtained  for  the  CS,  nreparatory  to  constructing  the  composite 
score,  factor  extension  techniques  can  be  used  to  show  how  the  PS  measures 
relate  to  the  CS  factors  (see  Appendix  II).  This  method  uses  the  cross 
correlations  between  the  PS  and  CS,  in  conjunction  with  the  CS  factor 
loadings,  to  comoute  the  loadings  of  the  PS  on  the  CS  factors.  The 
end  result  is  a factor  matrix  for  the  PS  where  the  factors  are  defined 
by  the  CS  analysis. 

This  procedure  permits  definition  of  the  PS  variables  in  terms  of 
broad,  uncorrelated  factors,  and  should  facilitate  theoretical  development 
by  permiting  unai;ibiguous  coupling  of  the  PS  and  a set  of  theoretical 

1 
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constructs  (factors).  It  should  be  noted  that  the  factorially 
complexity  of  neither  the  PS  nor  the  CS  is  of  paramount  importance. 

So  long  as  the  factor  solution  of  the  CS  is  interpretable  psychologically, 
the  procedure  is  workable  and  provides  substantive  information  about  the 
PS-CS  relationship. 

6 . Summary 

This  paper  has  reviewed  the  various  methods  of  criterion  construction 
under  conditions  of  single  and  multiple  criteria,  as  well  as  the  theoretical 
justification  for  these  procedures,  and  has  developed  a two  dimensional 
framework  for  conceptualizing  the  criterion  problem.  It  was  noted  that 
controversies  surrounding,  (1)  construct  validity,  and  (2)  the  multiple  vs 
single  criterion  camns  seem  to  be  of  a "straw-man"  nature.  Based  on  the 
review  and  the  development  of  the  theoretical  framework,  a solution  for 
the  composite  construction  problem  was  proposed  which  may  have  optimal 
properties.  Certain  implications  of  the  model  in  terms  of  interpretation 
and  factorial  complexity  of  the  PS  and  CS  measures  was  discussed. 


520 


TECHNICAL  APPENDIX  I 


For  the  following  develonment  define 

N:  the  number  of  observations;  sample  size  for  the  CS  and  PS 

h.  the  number  of  variables  in  the  CS 
p:  the  number  of  factors 

q;  the  number  of  variables  in  the  PS 
Let  R be  the  nxn,  correlation  matrix  among  the  CS  variables.  Compute  A, 
nxp,  which  is  the  common  factor  matrix  for  p factors  based  on  the 
canonical  factor  analysis  algorythm  (Rao,  1955),  such  that 

(1)  AA'=  U'^RU'^ 

2 

holds  in  an  approximate  sense.  U is  the  diagonal  matrix  of  unique 
variance  estimates.  This  is  the  usual  factor  analysis  model,  postulating 
a rank  p factor  matrix  to  annroxiniate  the  common  portion  of  the  data. 

The  canonical  algorythm  is  chosen  because  of  the  desirable  form  of  the 
resulting  factor  scores  (Harris,  1968).  However,  other  methods,  such 
as  princinal  factor  analysis,  could  be  used  with  little  imoact  on  the 
resulting  CC.  At  this  stage  we  generally  require  a transfonnation  on  A, 
say  T,  nxn,  wiiich  nrovides  a more  easily  internretable  solution.  Thus,  let 

(2)  A*=  AT 
and 

(3)  F=  ZR'^A*. 

F,  Nxp,  is  a set  of  factor  scores  estimated  using  the  so  called  regression 
method  (Hannon,  1967),  v/here  Z,  Nxn,  is  the  original  scaled  score  matrix 
such  that 

(4)  Vl=  R. 
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Now,  if  we  let  f . . ,i=l ,2,. . . ,N,j  = l ,2  . . . ,p,  be  the  factor  score  for 

* \j 

the  i-th  person  on  the  j-th  factor,  the  CC  variable,  C^. , is  defined 

(5) 

J-1 

where  W.  represents  the  sum  of  squared  factor  loadings  for  the  j-th 

\J 

factor  of  A*.  It  should  be  noted  that  F constitutes  a set  of 
scores  which  al^e  uncorrelated,  i.t. 

(6)  F'F  = D 

where  D is  a diagonal  matrix.  In  practice  F would  be  normalized  to 
unit  length  before  (5). 


TECHNICAL  APPENDIX  II 


Let  Rp^,  q X h,  be  the  cross  correlations  between  the  PS  and  the  CS. 
We  wish  to  find  a matrix  A such  that 

D 

(1)  ApA*'  = Rpc, 

i.e.,  Ap  is  a set  of  factor  loadings  for  the  PS  variables  on  the  CS 
factors.  The  solution  is  (Dwyer,  1937), 

(2)  Ap  = Rpc  A*(A*‘A*)-1. 
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APPLICATIONS  OF  PREDICTOR  ORDERING  AND  SELECTION 


BY  A BAYESIAN-DECISION  TECHNIQUE 
S.  E.  Bowser 

Navy  Personnel  Research  and  Development  Center 
San  Diego,  California 
Introduction 

The  selection  of  predictor  variables  has  been  the  pastime  of  researchers 
since  the  Inception  of  behavioral  science  research.  We  have  tried  to  find 
the  variables  which  are  the  "valid"  predictors  of  criteria  for  our  problems. 
In  a prediction  problem  there  usually  exists  a need  to  use  some  character  or 
variable  selection  process  to  reduce  the  number  of  variables  required  to  make 
■redlctlons  in  operational  situations.  The  concept  of  selection  engenders 
^he  idea  of  ordering  the  variables  so  that  the  "best"  subset  can  be  selected 
for  operational  utilization.  The  problems  associated  with  predictor  ordering 
and/or  selection  are  multidimensional. 

The  nature  of  the  ordering  and  selection  methodology  used  is  dependent 
upon  tlie  types  of  variables  being  considered.  The  analyses  areas  can  be 
diaracterized  by  the  nature  of  the  criteria  and  predictor  variables.  T'he 


Insert  Table  I 


type  A predictor  ordering  and  selection  problem  is  generally  handled  by 
iccretion-deletion  techniques  (Johnson,  1949,  p.  331)  within  the  general 
class  of  regression  problems.  Multiple  regression  is  commonly  used  in  person- 
nel research,  often  by  making  unwarrented  assumptions  about  the  variables 
being  used.  The  reason  behind  this  tendency  to  over  use  regression  methodology 
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lies  in  the  fact  that  the  technique  is  readily  available  in  most  modern 
computer  centers  and  other  techniques  are  not  as  available. 

The  type  B ordering  and  selection  problem  is  normally  dictated  by 
theory  and/or  practical  considerations.  It  so  happens  that  the  analysis  of 
variance  models  can  be  related  to  the  analysis  of  data  collected  by  design 
of  experiments  techniques  and  also  can  be  used  for  type  B prediction.  The 
analysis  of  variance  models  are  related  directly  to  the  multiple  regression 
models.  The  principal  difference  lies  in  the  independent  variables  being 
quantitative  in  type  A prediction  and  qualitative  in  type  B prediction.  The 
distinction  of  quantitative  versus  qualitative  is  often  within  the  control 
of  the  experlmentor  due  to  the  general  availability  of  scaling  methodology. 

The  type  C ordering  and  selection  is  defined  only  in  terms  of  theory 

and  logic,  not  methodd  rrcdlcrion  analysis  is  possible  through  Linear 

Discriminant  Function  (i  Liiei.,  U36).  In  specific  cases  the  generalized 
2 

distance  function  D of  Mahaln-.  bis  is  the  best  methodology.  In  general 
this  type  of  problem  is  converted  by  assumption  to  type  A and  the  ordering 
and  selection  done  vie  ul;at  moLliOdolo^y . 

The  type  D type  ^ ^ a;;’.  the  area  in  which  there  are  the  fewest 
number  of  methods  genfaJly  available.  In  terms  of  prediction  the  area  of 
information  theory  gives  general  guide  lines  but  there  is  no  specific  method- 
ology. Order  statistics  provide  methodology  for  prediction  analysis  but 
are  of  little  help  in  the  ordeiing  and  Si»’..ection  problem.  A new  approach  via 
Bayeslan-Declslon  theory  has  been  established.  The  computer  program  for  this 
methodology  is  titled  CHAROSEL  (Bowser,  1973a),  The  CHAROSEL  technique  de- 
veloped by  Bowser  and  Moonan  (1973)  is  based  upon  the  Attribute  Bayesian 
Classification  Decision  (ABCD)  technique  developed  by  Moonan  and  presented 
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Ir.  its  Ini  rial  form  to  this  conference  last  year. 

The  appliiation  of  the  ABCD  technique  to  type  D problems  is  as  follows. 

The  pr.ib.ii.  i ' i ty  of  the  observed  category  of  predictor  variable  arising  as  a 
result  of  sain[>ling  eacii  criterion  category  is  first  calculated.  By  application 
of  Bayes'  formula,  this  gives  rise  to  posterior  probabilities  for  each  sub- 
ject of  ilie  testing  sample.  These  probabilities  are  then  combined  with  a 
cost  matrix  leading  to  specific  probability  of  criterion  group  membership. 
Wliiitu-'ier  decision  minimizes  costs  of  misclassl f ication  becomes  the  predictive 
class  toi  the  sample  objects.  Such  a strategy  of  decision  is  known  as  a 
Bayes  1 au-I)ecislon  teciinique.  CHAROSKL  is  the  technique  for  selection  and 
ordering  of  categorical  predictor  variables  for  categorical  criterion  pre- 
diction problems. 

'lilt*  application  of  CdlAROSid.  to  type  D selection  and  ordering  problems 
wil]  be  (iresented  in  some  detail.  The  population  of  Interest  is  sampled  in 
order  to  obtain  a training  sample.  The  purpose  of  this  sample  is  to  estimate 
l he  pn  ball  j I i t ios  of  endorsement  of  each  level  of  each  predictor  variable 
I . .1  ‘ literion  category.  The  prediction  is  effected  by  using  a cost 

I . ■ ■ ! . it ication  mntiix  and  posterior  prohah  ilit ies  of  criterion  mem- 
P t.d,  , •iii.iifd  by  i-ndorsfiiieni  rnttos  nn<i  all  combined  via  Bayes'  theorem. 

Iii^-  t ' dicisiiKi  tible  i determined  hv  Haves'  theorem  probability, 

"■  '■■■'I  . . I . r , li  (osi  of  ii.is;  I ass  i 1 i ca  t i on  , and  minimum  ubioctive  function. 

d-  i.iii:’  .ire  then  assi'mb  led  into  a r.  ..  ir^n  (sble  whose  rows 
I ■ : n ..  norie.  of  membership  and  wiiose  < o 1 nmns  represent  pre- 

< • 1 : ‘ i-,e.i.bft  sh  i p 1 oi  membeis  ot  the  testing  suhsamplc. 

.1  iiip  .* 'lb  . . are  ev.pe.  ii  d !o  he  independent  for  cross 

’ n s .i.ai  il  :.  slonld  be  la.T’  enongli  lo  accurately  esti- 

d p r.  .t..ih  i ! i t I-  . the  CliAROdr.f  [in.p.r  a.-n  prodneos  a new  decision 

■ ' prcM  till  v.ii;  ih  il'  .uidod  to  tile  .'iidl'tion  i.'.r.tem.  fite  program 
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selects  the  "best"  variable  at  the  time  but  It  Is  an  accretion  system  only 
In  Its  present  form.  The  nature  of  the  objective  function  used  has  been 
discussed  earlier  by  William  Moonan  In  his  presentation. 

The  development  of  the  CHAROSEL  program  is  a breakthrough  in  that  pre- 
dictions of  qualitative  criterion  variables  from  knowledge  of  qualitative 
predictor  variables  is  not  possible.  The  ordering  problem  is  also  answered 
by  application  of  this  technique.  This  procedure  allows  the  utilization  of 
variables  with  no  assumptions  beirg  made  about  their  distribution  or  scaling 
properties.  The  only  assumption  that  is  made  is  that  the  predictor  variables 
are  independent  which  is  an  assumption  that  is  made  by  most  theories  when 
this  type  of  material  is  analyzed. 

Application 

The  areas  for  which  the  CHAROSEL  procedure  has  had  application  are 
both  diverse  and  interesting.  This  methodology  can  be  applied  to  any  sit- 
uation In  which  an  attempt  is  made  to  predict  categorical  criteria  from 
categorical  predictors.  In  most  cases  the  applications  have  dealt  with  non- 
cognitive  data  bases.  It  is  interesting  to  note  that  the  technique  is  not 
limited  to  behavioral  science  applications,  but  has'  had  impact  on  other 
areas  of  research,  Tlie  areas  of  potential  application  are  expanded  if  one 
just  observes  that  many  continuous  variables  are  naturally  categorical 
variables  which  have  been  scaled. 


Insert  Table  II 


An  overview  of  specific  applications  are  listed  in  table  II.  The 
first  two  items  listed  in  the  table  are  derived  from  the  project  Classifi- 
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cation  Prloi  to  Enlistment.  The  findings  reported  represent  initial  findings 
rather  than  cross-validated  results.  The  final  results  of  this  project  have 
recently  become  available  and  show  encouraging  results  in  the  cross-validation 
(Bowser,  1973b).  The  principal  point  here  is  that  the  criteria  are  of  great 
Interest  to  the  military  researcher  and  the  data  source  is  relatively  untapped. 
The  non-cognitive  data  bases  represent  a broad  area  of  reliable  predictor 
variables  which  are  easy  to  obtain. 

The  next  item  on  the  table  is  a non-military  project  conducted  in  a 
welfare  agency  on  skid  row  In  Los  Angelos  (Bowser,  Sherman,  & Whlsler,  1974). 
The  results  reported  in  the  table  are  an  attempt  to  look  at  whether  or  not 
non-cognitive  factors  could  be  used  for  screening  successful  from  unsuccessful 
individuals  in  this  type  of  project.  I t is  apparent  that  biographical  data  has 
potential  for  this  type  of  popul'tion  ns  a source  of  prediction  variables 
(Bowser,  in  preparation). 

The  target  classification  research  is  the  next  item  on  the  list.  Tliis 
project  understandably  has  great  importance  to  the  Navy.  The  details  of  the 
variables  used  are  not  available  but  tlie  excellent  results  can  be  pointed 
out.  This  project  demonstrates  the  versatility  of  application  this  method- 
ology. It  is  a good  example  of  a non-behavloral  science  data  base  to  which 
this  procedure  has  been  successfully  applied.  It  is  also  obviously  an  ex- 
cellent example  of  the  advantage  of  being  able  to  utilize  cost  of  misclassi- 
flcation.  In  the  target  classification  r cost  of  misidentl  f ication  of 
foreign  submarines  as  friendly  is  mucli  greater  than  tlie  cost  of  classifying 
our  submarines  as  surface  ships.  It  is  rhus  restated  that  the  potential  of 
CHAROSEL  to  directly  use  a cost  matrix  is  of  a definite  advantage. 
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The  next  Item  on  our  list  is  a good  example  of  the  type  of  application 


which  usually  would  have  been  pushed  into  the  regression  mold.  The  flex- 
ibility questionnaire  is  the  type  of  instrument  that  is  usually  scaled  into 
a continuous  variable.  The  other  approach  is  to  break  it  down  into  zero-one 
data  and  apply  regression  analysis.  The  results  reported  in  the  table  show 
that  when  utilizing  CHAROSEL  the  prediction  of  the  criteria  is  as  good  or 
better  than  the  expected  results  from  the  more  traditional  analysis  procedures. 

Considering  page  two  of  table  II,  the  first  item  is  interesting  in 
that  the  criteria  is  polychotamous , a difficult  situation  for  linear  regres- 
sion to  handle.  The  point  is  that  the  CHAROSEL  technique  handles  this 
problem  with  relative  ease.  The  results  reported  here  are  in  this  case 
preliminary,  as  this  project  is  continuing  at  the  present  time.  This  limited 
application  was  utilized  to  verify  some  theoretical  ideas  as  to  the  questions 
which  were  Important  to  this  prediction  problem;  a definately  useful  approach 
for  this  technique. 

The  application  of  this  methodology  to  a standard  test  such  as  the  Strong 
Vocational  Interest  Blank  is  the  next  item  on  our  list.  The  use  of  new  methods 
on  old  problems  is,  or  at  least  can  be,  a good  thing.  It  sometimes  allows 
a new  viewpoint  to  be  established.  In  the  case  reported  here  the  results 
are  encouraging.  Please  note  the  maximum  correct  decision  rate  which  is 
95.4%.  This  result  is  not  cross-validated  and  so  should  be  viewed  with 
some  caution  but  it  is  still  very  good  and  demonstrates  a real  promise  for 
applying  tliis  method  to  established  test  instruments. 

In  the  next  application,  the  criteria  was  artificially  categorized. 

Since  the  usual  case  is  in  the  opposite  direction  this  is  somewhat  interesting. 
Tlie  results  as  obtained  are  quite  gratifying.  The  particular  problem  of 
reserve  retention  addressed  here  is  one.  of  great  Interest  to  all  military 
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researchers.  Again  this  method  may  allow  the  use  of  data  sources  previously 
Ignored  or  not  recognized  as  demonstrated  in  this  case. 


The  identification  of  gifted  children  Is  the  last  project  on  our  list. 

It  gives  us  a clue  to  potential  test  construction  approaches  that  are  less 
culturally  biased  than  our  present  methodologies.  It  should  be  noted  that 
this  study  was  conducted  with  a population  consisting  of  children  princi- 
pally from  minority  groups.  This  study  was  reported  In  more  detail  by 
William  Moonan  in  this  proceeding. 

The  last  Item  on  this  table  demonstrates  an  approach  which  obviously 
has  great  potential  value.  Of  course  the  details  of  the  predictor  variables 
are  unavailable  but  the  results  are  impressive.  If  nothing  else,  the  imple- 
mentation of  these  results  will  reduce  the  amount  of  time  needed  to  board 
commercial  airliners,  a problem  I'm  sure  is  of  interest  to  all  those  here. 

As  has  been  stated  the  actual  applications  have  covered  a broad  range. 
Tlie  obvious  versatility  of  this  methodology  demonstrates  its  great  potential 
to  researchers,  Ttie  applications  to  date  have  Included  military  as  well 
as  eivili.in  les.arch  problems.  The  problem  areas  have  been  both  in  and  out 
of  the  beliavior.il  sciences.  The  data  bases  utilized  have  Included  sonar 
clues,  biogr.iphlc.il  inventories,  and  vocational  interest  inventories. 

Potential  Applications 

It  is  fairly  obvious  from  the  general  applications  already  utilized 
that  new  .applications  arc  almost  unlimited.  Areas  of  application  that 
readily  come  to  mind  are  those  of  the  medical  field.  The  potential  of  this 
technique  as  a diagnostic  tool  is  apparent.  With  further  thought  we  can  also 
see  the  possibilities  of  utilizing  this  metliod  for  diagnostic  syndrome 
detection. 
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The  possible  applications  to  this  area  could  take  many  forms,  such  as 
the  research  ideas  just  stated,  or  in  the  more  direct  application  of  provi- 
ding a physician  with  probabilities  of  diagnostic  category.  Since  the 
medical  profession  is  currently  using  physician's  assistants  to  help  reduce 
the  work  load  for  the  doctors,  any  technology  which  helps  to  objectify 
diagnostic  procedures  has  great  value. 

The  military  applications  already  reported  include  the  area  of  recruit 
f.creenlng  and  selection.  An  expansion  in  this  area  of  application  could 
possibly  be  made  in  the  area  of  prediction  of  career  field  success.  Since 
the  studies  in  this  area  have  utilized  biographical  information,  it  is  also 
I'Ossible  as  a byproduct  of  this  research  to  develop  a more  complete  bio- 
graphical profile  of  the  successful  military  recruit.  Investigation  of  such 
profile  would  allow  the  military  services  to  gain  a better  understanding 
f the  types  of  Individuals  who  are  successful  in  military  service.  The 
understanding  of  the  individual  types  which  are  successful  would  allow  more 
; easonable  manpower  planning  and  provide  the  basis  for  modeling  proposed 
Iterations  in  screening  techniques. 

Tlie  area  of  organizational  development  and  structure  could  also  be 
• ffected  by  application  of  this  technology.  While  it  is  not  obvious,  it 
is  possible  to  use  this  methodology  to  uncover  managerial  value  systems. 
Another  related  area  lies  within  the  realm  of  implicit  judgemental  criteria, 
for  instance,  personnel  rating  systems.  It  is  further  possible,  by  utilizing 
personal  probability  estimates,  to  use  this  method  in  obtaining  data  concern- 
'"ng  decision  processes  themselves.  This  of  course  has  great  Importance  to 
managerial  systems.  Perhaps  another  area  which  is  related  to  the  foregoing 
should  be  explored,  that  is,  the  determination  of  Implicit  versus  explicit 
goals  of  an  organization. 
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While  this  technology  is  primarily  concerned  with  categorical  information 
as  source  material,  it  is  recognized  that  mixtures  of  continuous  and  cat- 
egorical information  are  likely. ‘ One  possibility  for  handling  this  situa- 
tion using  the  present  computer  program  would  be  simply  to  categorize  the 
continuous  variables.  This  procedure,  while  the  reverse  of  what  is  usually 
done,  has  certain  features  of  arbitrariness  which  make  it  undesirable.  It 
is  possible  to  obtain  probabilities  from  individual  continuous  scores.  The 
method  utilizes  the  mathematical  features  of  the  density  function.  There 
has  been  a program  completed  called  MIX,  (Moonan  & Covher,  1972)  which 
was  developed  by  William  Moonan;  this  program  does  combine  categorical  and 
continuous  variables  in  order  to  obtain  a decision  table.  It  should  be 
noted  that  this  procedure  is  not  comparable  to  CHAROSEL  since  it  does  not 
have  the  selection  and  ordering  feature.  It  is  possible  to  adapt  CHAROSEL 
to  use  both  continuous  and  categorical  predictor  variables.  The  proposed 
new  program,  CHAROSEL  II,  would  broaden  the  applicability  of  this  technique 
considerably.  Other  alterations  of  this  approach  are  potentially  possible, 
such  as  altering  the  objective  function  in  order  to  obtain  items  which  were 
the  least  discriminator".  The  finding  of  items  with  great  communality  in 
some  circumstances  is  more  important  than  finding  those  items  wliich  discriminate. 
It  is  possible  to  weight  the  objective  function  so  that  a given  class  of 
prediction  is  made  more  important.  Tliis  is  over  and  above  the  use  of  a 
cost  of  mi  scl  assi  f 1 cat  i on  m.itrix. 

Tlie  potential  applications  of  the  present  system  are  extensive.  The 
system  obviously  can  be  expanded.  Tlie  possibility  is  there  to  broaden  the 
applications  of  this  technique  into  the  interface  area  between  type  C and 
type  D prediction  problems.  Tlie  actual  expansion  of  this  technique  into 
its  many  potential  applications  will  of  course  depend  upon  the  information 
of  its  availability  reaching  the  appronriatc  researchers. 


CONCLUSIONS 


This  technique  Is  Indeed  a breakthrough  in  the  area  of  predictor  ordering 
and  selection  techniques.  It  provides  ready  access  for  the  researcher  into 
vast  new  data  sources  previously  unavailable.  The  fact  of  the  matter  is 
that  with  the  present  anti-test  pressures  that  are  being  exerted,  new  data 
sources  are  virtually  imperative.  The  technique  as  described  offers  a tech- 
nology capable  of  utilizing  the  non-cognltive  data  bases. 

The  method,  while  offering  promise,  still  has  drawbacks.  It  is  not 
readily  usable  without  the  aid  of  large-scale  computers.  Operational  systems 
based  upon  this  technique  would  require  computer  support.  Another  problem, 
not  quite  so  obvious,  is  the  fact  that  this  system  lacks  intuitive  appeal. 

The  logic  of  the  procedures  used  is  not  readily  obvious  to  individuals  who 
are  not  familiar  with  Bayesian  inferential  systems.  Given  these  drawbacks 
and  possibly  others  not  yet  recognized,  this  method  is  still  a promising 
new  approach. 

It  is  hoped  that  the  introduction  of  this  new  approach  will  lead  to 
broad  new  applications.  The  technique  itself  obviously  needs  to  be  explored 
in  more  depth.  It  can  be  shown  that  this  technique  is  the  same  as  linear 
regression  if  the  assumptions  are  met.  However,  since  no  linear  assumptions 
are  necessary  for  this  approach  it  is  assumed  that  it  is  applicable  where 
the  linear  model  is  inappropriate.  It  is  Intuitively  obvious  that  nature 
does  not  always  proscribe  the  linear  model.  It  is  hoped  that  this  procedure 
will  be  explored  more  completely  for  its  theoretical  properties.  Meanwhile 
perhaps  the  application  of  CHAROSEL  will  give  added  dimension  to  the  forma- 
tion of  predictive  systems  and  theories. 
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GROWING  DEMANDS  ON  HUMAN  RESOURCES: 

A VIEW  FROM  INDUSTRY 

Thomas  C.  Tuttle 

Westinghouse  Behavioral  Safety  Center 
Columbia,  Maryland 

Not  too  many  years  ago,  when  I was  fresh  out  of  college 
serving  my  tour  of  duty  in  the  Army,  it  was  customary  to  speak  of 
everything  beyond  the  scope  of  the  Defense  Department  as  "the  out- 
side." In  those  days,  those  of  us  who  did  not  intend  to  make  a 
career  out  of  the  service  would  talk  of  how  things  would  be  dif- 
ferent once  we  got  out.  Now  that  I'm  standing  on  the  outside, 
looking  at  the  theme  of  this  conference,  I'm  not  sure  that  the 
differences  are  as  great  as  I had  imagined.  Certainly,  with  re- 
gard to  the  topic  "Growing  Demands  on  Human  Resources"  there  is 
considerable  similarity.  Therefore,  I welcome  this  o^^portunity 
to  look  at  the  situation  from  the  point  of  view  of  industry  and 
to  discuss  some  possible  behavioral  science  responses  to  these 
now  and  increasing  demands  on  human  resources. 

Perhaps  a more  accurate  way  to  express  the  view  from  industry 
would  be  to  use  the  expression  "Growing  Demands  on  Human  Systems" 

The  recent  HEW  Special  Task  Force  concerning  Work  in  America,  (1972)  , 
pointed  out  that  in  many  areas  corporate  responsibility  is  being 
expanded  into  areas  that  traditionally  have  received  little  industry 
attention.  Increasingly  the  corporate  balance  sheet  is  being  asked 
by  society  through  legislation  and  social  pressure  to  reflect  some  of 
the  costs  of  doing  business  that  the  larger  society  has  had  to  bear. 
This  trend  is  reflected  in  the  increasing  concern  of  organizations 
with  developing  systems  for  human  resource  accounting. 
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There  are  a number  of  areas  in  which  this  increasing  responsi- 
bility is  occurring:  (1)  worker  health,  (2)  worker  safety,  (3)  quality 
of  work,  (4)  environmental  quality,  (5)  employment  of  women,  blacks, 
and  other  minority  groups.  Each  of  these  areas  of  corporate  respon- 
sibility has  generated  demands  on  organizations  to  develop  structures, 
policies,  and  methodologies  to  deal  with  these  new  problems.  Thus, 
not  only  are  there  growing  demands  on  the  human  resources  in  organiza- 
tions, there  are  increased  demands  on  the  organizational  systems 
themselves.  Unless  these  systems  become  more  adaptive,  it  will  be 
difficult  for  them  to  make  most  effective  use  of  their  human  resources 
In  this  paper  I would  like  to  describe  two  areas  of  increasing 
corporate  responsibility  in  some  detail,  and  discuss  a long-range 
behavioral  approach  to  these  problems  being  developed  and  tested  by 
the  Westinghouse  Behavioral  Safety  Center.  The  two  areas  to  be  dis- 
cussed are  (l)the  quality  of  work  life,  and  a closely  related  area 
of  concern,  (2)  occupational  safety.  Both  of  these  areas  are  re- 
ceiving increased  attention  by  agencies  in  the  Federal  Government, 
such  as  the  National  Institute  of  Occupational  Safety  and  Health, 
the  Occupational  Safety  and  Health  Administration  and  the  National 
Commission  on  Productivity. 

Quality  of  Work 

"Quality  of  Work"  or  "Quality  of  Work  Life"  are  terms  that 
encompass  efforts  to  redesign  jobs  and  restructure  organizations  to 
provide  increased  positive  outcomes  for  workers,  unions  and  management 
The  concept  as  defined  by  the  National  Commission  On  Productivity 
(Mills,  1973)  includes  all  activities  associated  with  such  terms  as: 
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j ob  CUT  ')  chinon t. 
job  satisfaction 
work  motivation 
job  redes itjn 
humanization  of  work 
worker  participation 
organizational  development 
socio-technical  development 

Anyone  who  reads  his  local  newspaper  or  the  weekly  news  maga- 
zines is  familiar  with  press  reports  concerning  the  declining  quali- 
ty of  work.  The  terms  Chwngiyuj  Wci'K  BZue  — CoZZuv  BZubs  and 

White-CoZZar  Woes,  have  become  part  of  our  vocabulary.  One  frequent- 
ly cited  illustration  of  this  problem  is  the  situation  at  the  General 
Motors  Assembly  Plant  in  Lordstown,  Ohio.  This  plant,  equipment  and 
work  procedures  were  designed  as  a model  of  efficiency.  However,  it 
neglected  one  crucially  important  consideration,  the  effects  of  this 
engineering  marvel  on  the  workers.  The  work  force  at  the  nev;  Lords- 
town plant  was  predominantly  young  and  not  inclined  to  accept  the 
regimen  imposed  by  a super-autom.ated  assembly  line.  In  expressing 
their  discontent  to  this  high  speed  production  philoso’^hy,  the  workers 
"responded  with  absenteeism,  sabotage,  and  finally  a bitter  twenty- 
two  day  strike."  (Jenkins,  1973).  Similar  reactions  have  been  re- 
ported in  varying  degrees  across  the  country. 

Workers  entering  the  work  force  today  are  characterized  by 
higher  education,  higher  expectations,  and  the  demand  for  some 
sense  of  control  over  decisions  that  effect  their  lives.  As  a 
result,  new  workers  arc  less  willing  to  be  bossed  around  and  to 
perform  monotonous,  dead-end  jobs.  As  one  personnel  expert  stated, 

"It  is  becoming  increasingly  clear  that  the  employees 
entering  the  v;orld  of  business  are  unwilling  to  accept  the 
system  of  unanswered  questions,  half-baked  opportunities, 

and  half-assed  jobs They  v;ill  no  longer  unques tioningly 

arcej't  the  organizational  party  line  and  will  no  longer  stand 
idly  by  while  their  business  lives  are  controlled  and  manipu- 
\ it'‘d  from  above."  (Jenkins,  1973  , p.  78-79) 
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The  theme  of  worker  alienation  in  industry  has  even  been  picked 

up  in  a recent  Axmy  recruiting  poster  that  says  coaxingly,  "If  your 

job  puts  you  to  sleep,  try  one  of  ours." 

The  problem,  however,  goes  beyond  simply  dissatisfaction  with 

jobs.  Workers  want  involvement  and  a share  of  the  responsibility 

for  decision  making.  Some  propose  job  enrichment  or  job  enlargement 

as  a solution.  Others  argue  that  the  problem  goes  much  beyond  the 

rearrangement  of  work  tasks,  and  involves  the  whole  structure  of 

organizations.  This  latter  view  is  one  expressed  in  the  following 

quote  from  a writer  advocating  "Democracy  in  the  Factory- " 

" the  basic  problem  is  not  a faulty  arrangement  of 

jobs,  it  is  a question  of  faulty  power  patterns;  the 
arrangements  of  jobs  are  only  surface  symptoms.  To  be 
sure,  any  job  reshuffling  that  gives  workers  increased 
control  is  a step  on  the  road  to  democracy.  And  the 
assembly-line  worker  who  sees  the  time  span  of  his  job 
increased  from,  say,  forty  to  seventy  seconds,  and  his 
task  widened  from  tightening  one  bolt  to  tightening  two 
bolts  will  doubtless  appreciate  the  change.  But  strictly 
speaking,  only  those  organizational  systems  that  regard 
employee  influence  as  a healthy  and  desirable  phenomenon, 
and  that  are  planned  from  the  ground  up  to  nurture  and 
encourage  such  influence,  can  be  called  democratic." 

(Jenkins,  1973,  p.  83), 

A comprehensive  approach  to  improving  the  quality  of  work  must 
go  beyond  a concern  with  job  and  task  considerations,  although  they 
are  very  important,  and  must  consider  more  fundamental  organizational 
characteristics  as  well. 


Occupational  Safety 

The  problems  of  occupational  safety  have  been  highlighted  by 
the  passage  in  1970  of  the  Williams-Steiger  Occupational  Safety  and 
Health  Act,  and  the  1972  President's  Report  on  Occupational  Safety 
and  Health.  The  President's  Report  on  Occupational  Safety  and  Health 
(1972)  documents  the  fact  that  between  1961  and  1970  the  injury 
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frequency  rate  for  manufacturing  industries  (i.e.  the  number  of 
disabling  injuries  per  million  employee-hours  worked)  has  increased 
from  11.8  to  15.2,  an  increase  of  approximately  29%  in  ten  years. 

This  report  also  presents  data  resulting  from  safety  inspections 
conducted  in  1971  under  the  provisions  of  the  Williams-Steiger 
Occupational  Safety  and  Health  Act.  Out  of  13,174  inspections 
conducted  there  were  31,053  reported  violations.  Although  many  of 
these  reported  violations  were  not  in  the  serious  category,  they 
provide  some  indication  of  the  general  climate  of  the  organizations 
inspected  with  regard  to  safety  philosophy. 

In  terms  of  the  number  of  employed  persons  who  experience  health 
or  safety  problems  related  to  employment  there  are  no  firm  statistics. 
Nevertheless,  the  National  Safety  Council  (President's  Report  on 
Occupational  Safety  and  Health,  1972)  estimates  that  of  the  80 
million  employed  persons  in  the  civilian  labor  force,  over  14,000 
suffer  fatal  accidents  and  2.2  million  suffer  disabling  injuries  each 
year.  It  should  be  emphasized  that  these  numbers  are  only  estimates. 
They  do  not  attempt  to  account  for  the  unknovm  number  of  non-disabling 
accidents  and  illnesses  resulting  from  exposure  to  hazards. 

An  estimate  of  the  tangible  costs  resulting  from  industrial 
accidents  can  be  derived  from  the  expenses  incurred  by  employers 
in  the  form  of  workman's  compensation.  The  Social  Security  Admini- 
stration estimates  that  the  total  cost  of  the  Workman's  Compensation 
to  employees  in  1970  was  4.82  billion  dollars.  (President's  Report 
on  Occupational  Safety  and  Health,  1972)  Even  this  figure  is  an 
underestimate  of  the  actual  cost,  due  to  the  fact  that  many  workers 
are  not  covered  by  the  act.  In  addition  the  figure  does  not  include 
costs  due  to  retraining,  rehabilitation  necessitated  by  disabling 
accidents,  as  well  as  less  tangible  costs  such  as  emotional  distur- 
bance, and  disturbances  in  job  performance  or  home  life.  These  indices 
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substantiate  the  fact  that  accidents  and  illnesses  attributable 
to  the  work  environment  remain  a serious  problem. 

Relationship  between  Quality  of  Work  and  Safety 

Occupational  Safety  and  quality  of  work  life  are  both  inti- 
mately related  in  terms  of  their  impact  on  the  employee  and  in 
terms  of  organizational  factors  that  effect  them.  Certainly  a 
worker  who  must  face  an  extremely  hazardous  job  on  a daily  basis 
cannot  be  considered  to  have  high  quality  work  life.  Further- 
more, organizations  that  are  attentive  to  the  safety  hazards 
effecting  workers  are  also  likely  to  pay  attention  to  the  effects 
of  other  organizational  conditions  on  employee  satisfaction  and 
performance.  Therefore,  occupational  safety  concerns  can  be  con- 
sidered to  be  one  aspect  of  the  overall  quality  of  work.  Organi- 
zational factors  that  impact  positively  on  quality  of  work  criteria 
such  as  productivity,  satisfaction,  absenteeism,  and  turnover  can 
also  be  expected  to  have  a positive  impact  on  safety  criteria. 

Organizational  Development  Approach 

The  most  popular,  recent  approach  for  dealing  with  such  human 
problems  in  rndustrial  organizations  has  been  that  of  the  organiza- 
tional development  specialists.  Many  of  these  so-called  OD  special- 
ists have  claimed  considerable  success  in  dealing  with  problems  of 
worker  alienation,  declining  productivity,  high  absenteeism  and 
rapid  turnover.  The  size  of  tlie  consulting  fees  commanded  by  some 
of  these  OD  "super-stars"  represents  one  measure  of  their  success. 
Despite  the  fact  that  OD  specialists  have  undoubtedly  had  a positive 
impact  on  a number  of  organizations,  the  approach  as  a pragmatic 
answer  to  problems  throughout  industry'  seems  to  suffer  from  two 
major  weaknesses. 


First,  in  many  ways  the  practice  of  OD  is  more  of  an  "art" 
than  a science  at  this  point.  The  OD  specialist  typically  collects 
tv;o  types  of  information,  subjective  impressions  from  observations 
and  interviews,  and  objective  survey  data.  From  this  data,  diagnoses 
are  made  and  recommendations  for  action  formulated,  often  in  consulta- 
tion with  organizational  personnel.  As  a result  of  tlie  subjectivity 
of  much  of  the  data  and  the  diagnostic  process,  itself,  it  is  diffi- 
cult for  a specialist  to  communicate  to  someone  else  what  he  has 
done.  As  a result  the  approach  lacks  rigor  and  replicability. 

A second  weakness  is  that  there  just  aren't  enough  OD  special- 
ists available  to  deal  with  the  problems  facing  the  industrial  com- 
munity. The  process  is  quite  time-consuming  from  the  point  of  view 
of  the  consultant,  therefore,  one  individual  can  consult  with  a rela- 
tively small  number  of  organizations  in  a period  of  a year.  Further- 
more, training  of  effective  OD  consultants  is  complicated  by  the 
vagueness  surrounding  the  behaviors  required  to  successfully  operate 
in  the  organizational  setting.  As  a result  it  seems  unlikely  that 
Organizational  Development  specialists  can  be  trained  in  adequate 
numbers  to  meet  the  demands  placed  on  them. 

Westinghouse  Behavioral  Safety  Center  Approach 

How  then  will  organizations  be  able  to  cope  with  the  growing 
demands  placed  upon  them?  What  methodologies  can  be  used  to  improve 
the  quality  of  work  life  for  management,  unions,  and  employees? 
Obviously  there  are  many  answers  to  such  a question.  Just  as  obviously, 
there  are  probably  none  that  are  entirely  satisfactory.  Nevertheless, 

I v;ould  like  to  discuss  an  approach  currently  being  developed  by  the 
Westinghouse  Behavioral  Safety  Center  which  we  feel  has  considerable 
promise. 
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This  approach  grew  out  of  in-house  research  as  well  as  re- 
search conducted  under  contract  to  the  National  Institute  of 
Occupational  Safety  and  Health  (NIOSH).^  The  approach  recognizes 
that  organizations,  whether  they  are  military,  private  industry, 
government  or  educational,  are  complex.  Thus,  any  approach  must 
consider  a wide  range  of  variables  in  order  to  include  those 
variables  that  effect  quality  of  work  or  safety  outcomes.  The 
first  step  of  this  research  program  involved  development  of  a set 
of  behavioral  guidelines  by  surveying  research  literature  in  four 
areas  of  psychology:  (1)  organizational  psychology,  (2)  behavior 

modification,  (3)  training  and  (4)  engineering  psychology.  These 
guidelines  reflect  the  state  of  the  art  in  each  of  these  four  areas 
with  primary  regard  for  variables  that  have  implications  for  acci- 
dent reduction.  However,  as  discussed  further  below,  the  guidelines 
have  broader  applicability  for  other  organizational  outcomes.  Each 
guideline  consists  of  a short  definition,  followed  by  a 1-3  page 
description  of  the  supporting  research  literature  and  the  implications 
of  the  guidelines  for  organizational  functioning.  Approximately  80 
guidelines  were  defined,  distributed  approximately  equally  across  the 
four  research  areas.  The  guidelines  were  written  primarily  for  the 
safety  professionals,  rather  than  for  behavioral  scientists,  thus  they 
may  appear  to  be  somewhat  oversimplified. 


I Westinghouse  Behavioral  Safety  Center  contracts  with  NIOSH  include: 

HSM  99-72-27  Psychological  Behavioral  Strategy  for  Accident  Control 

HSM  99-73-61  Implementation  of  Psychological  Behavioral  Strategy 
for  Accident  Reduction 
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Examples  of  selected  guideline  definitions  are  presented 
below.  Included  are  two  guideline  definitions  from  each  of 
the  four  research  areas.  The  complete  outlines  giving  the 
titles  for  all  the  guidelines  in  each  of  the  four  research 
areas  appear  in  Appendix  A. 

1.  Organizational  Psychology  (Dachler  and  Schneider,  1973) 

a.  Maintaining  group  norms.  Group  pressure  is  an 
effective  means  for  maintaining  behavior  in 
accordance  with  group  norms  that  encourage  safety. 
Organizations  can  enhance  the  effectiveness  of 
group  pressure  by  (1)  creating  work  groups  that 
are  attractive  to  workers,  (2)  promoting  cooper- 
ation among  work  groups  and  (3)  by  encouraging 
social  contact  within  the  v/orkp2ace. 

b.  Job  design.  Jobs  should  be  designed  to  enhance 
worker  interest  and  involvement.  Jobs  which  are 
designed  to  emphasize  intrinsic  factors  (e.  £•, 
recognition,  variety,  autonomy,  responsibility) 
tend  to  contribute  to  higher  worker  satisfaction 
and  improved  v/ork  quality 
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2. 


Behavior  Modification  (M.-Tntir<-  and  White,  1973) 


a.  Social  reinforcerpont-  - !:ur  rvi.  cry.  Social  re- 

actions from  supervisor.-  ruch  as  praise,  attention, 
ridicule,  laughter,  or  facial  gestures  are  effective 
consequences  of  behavior.  They  can  be  used  to  in- 
creai  e the  frequency  of  behavior  which  promotes 
safety  and  to  decrease  the  frequency  of  behavior 
which  contributes  to  accidents. 

b.  Non- con tin gent  reinforcement.  If  re in forcers  are 
not  given  in  response  to  syecific  behaviors  and 
according  to  an  explicit  and  predictable  pattern, 
they  cannot  be  expected  to  increase  the  likelihood 
of  desired  safety  behavior. 

3.  Training  (Goldstein,  1973) 

a.  Feedback.  Knowledge  of  results  serves  a motiva- 
tional function  and  a guidance  function.  As  a 
general  rule,  training  aids  should  be  designed 
to  provide  maximum  and  imm.ediate  feedback  to  the 
trainee  as  to  the  adequacy  of  his  safety  perfor- 
mance. It  is  further  important  to  determine  that 
the  feedback  is  being  used  by  the  trainee.  For 
learning  purposes  it  is  useless  to  present  the 
trainee  w’ith  feedback  information  which  he  cannot 
understand. 

b.  Overlearning.  Tasks  or  procedures  that  may  have  to 
be  performed  in  emergencies  or  other  stressful  situa- 
tions should  be  practiced  until  they  become  overlearned. 
Such  overlearning  incrcasc.s  the  resistance  of  task  per- 
formance to  disruption  or  interference  under  stress. 
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4. 


Engineering  (Grether,  1973) 


a.  Job  safety  design.  Organizations  should  ensure  that 
jobs  are  designed  to  minimize  human-initiated  errors 
and  accidents.  Specifically,  jobs  should  be  comprised 
of  related  sets  of  task  activities  that  are  compatible 
with  each  other  in  terms  of  task  sequencing,  time  utili- 
zation, location,  and  function.  The  tasks  should  be 
logically  grouped  and  allocated  to  either  individuals 

or  work  teams. 

b.  Work/rest  cycle.  In  establishing  policies  relating  to 
work/rest  period  scheduling,  organizations  should  ensure 
that  (1)  the  work  duration  does  not  exceed  four  consecu- 
tive hours,  (2)  the  duration  of  rest  periods  permits 
physiological  and  cognitive  recovery  from  task  demands, 

(3)  rest  periods  are  scheduled  throughout  the  day  or 
work  period  in  accordance  with  the  level  of  environ- 
mental or  task  demands,  and  (4)  scheduling  and  duration 

of  rest  periods  allowed  for  social  contact  with  co-workers. 

The  full-set  of  behavioral  guidelines  represents  the  knowledae  base 
for  the  long-term  approach.  In  addition  the  guidelines  provide  some 
immediate  payoff  since  they  give  the  practitioner  a summary  in  rela- 
tively simple  terms,  of  some  relevant  behavioral  science  principles. 

This  summary,  although  not  an  answer  to  all  his  problems,  can  be 
used  to  assist  him  in  thinking  about  safety  problems,  instructional 
problems,  or  any  of  a wide  range  of  human  resources  problems  he  may 
face . 
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Another  key  ingredient  of  the  approach  is  represented  by 
the  Organizational  Performance  Intervention  Model  (Appendix  B) . 

This  model  serves  to  identify  the  major  categories  of  variable  s 
assumed  to  be  relevant  to  quality  of  work  and  safety  criteria, 
and  provides  a means  for  integrating  a wide  range  of  variables 
into  a single  approach.  It  also  serves  to  make  explicit  some  of 
the  assumptions  underlying  this  research  progreim. 

During  the  development  of  this  model,  it  became  obvious  that 
the  original  assumption  of  the  approach  was  valid.  This  assumption 
held  that  the  variables  which  effect  the  level  of  safety  in  the 
organization,  also  effect  other  outcomes  such  as  job  satisfaction, 
productivity  and  overall  "quality  of  work."  Thus,  the  guidelines 
developed  for  safety , can  with  very  little  modification,  be  considered 
to  be  guidelines  for  improving  overall  quality  of  work.  For  example, 
job  design  is  an  important  variable  with  regard  to  safety  as  well  as 
for  quality  of  work  criteria  such  as  job  satisfaction  or  productivity. 
Jobs  that  are  composed  of  imcompatible  tasks  or  which  lead  to  condi- 
tions of  information  overload  are  likely  also  to  lead  to  high  levels 
of  stress,  low  satisfaction  and  increased  likelihood  of  accidents. 
Despite  the  fact  that  the  guidelines  were  developed  in  the  context 
of  occupational  safety,  their  generality  is  not  restricted  to  safety 
concerns . 

The  model  and  the  set  of  behavioral  guidelines  form  a background 
for  the  Westinghouse  long-range, quality  of  work  research  program. 

This  background  provides  a firm  base  from  which  to  launch  organiza- 
tional intervention  programs.  The  intervention  strategy  is  character- 
ized by  three  phases:  (1)  Organizational  Diagnoses,  (2)  Implementation 

and  (3)  Evaluation. 
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The  organizational  diagnostic  procedure  attempts  to  avoid  the 
lack  of  rigor  inherent  in  the  typical  organizational  development 
type  of  diagnosis.  One  aspect  of  the  diagnostic  procedure  includes 
a diagnostic  inventory  to  be  completed  by  employees.  The  diagnostic 
inventory  is  based  on  the  organizational  model  discussed  earlier  and 
includes  items  designed  to  measure  the  following  sets  of  variables; 

(1)  organizational  characteristics  (e.g.,  structure,  decision  making, 
policies,  etc.)  (2)  supervisor  and  work  group  characteristics 

(e.g.,  supervisory  style  and  competence,  competition,  co-operation, 
etc.)  (3)  job  and  task  characteristics  (e.g.,  role  ambiguity, 
information  overload,  variety,  etc.)  (4)  individual  attitudes  (e.g., 
dimensions  of  satisfaction)  (5)  characteristics  of  the  work  environ- 
ment (e.g.,  lighting,  temperature,  adequacy  of  supplies,  etc.).  In 
addition  to  the  diagnostic  inventory  the  diagnostic  procedure  also 
includes  provision  for  a company  records  audit.  This  procedure 
will  yield  criterion  information  on  variables  such  as:  (1)  productivity, 

(2)  absenteeism,  (3)  turnover,  (4)  accidents,  and  (5)  grievances. 

The  criterion  information  is  needed  in  order  to  establish  statistical 
relationships  between  organizational  process  variables  and  individual 
and  organizational  outcomes.  Furthermore  it  will  provide  baseline 
data  for  the  evaluation  phase. 

Implementation 

The  overall  diagnosis  yields  an  organizational  profile  along 
a number  of  relevant  dimensions.  This  is  useful  to  the  extent  that 
it  allows  for  recommendations  for  action  that  are  tailored  to  the 
needs  of  the  organization.  The  mechanism  for  bridging  the  gap  between 
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the  diagnosis  and  intervention  is  based  on  a set  of  action  modules 
and  a tracing  procedure  for  linking  the  action  modules  to  the  diag- 
nostic procedure. 

The  action  modules  are  sets  of  actions  that  an  organization 
can  take  to  accomplish  some  specific  objective.  These  modules 
are  based  on  the  previously  discussed  behavioral  guidelines,  and 
represent  essentially  a translation  of  the  guidelines  into  action 
terms.  Each  module  is  oriented  toward  some  specific  objective  such 
as  implementation  of  a supervisory  reward  system,  development  of 
a safety  training  program,  implementation  of  a behavioral  record 
keeping  system,  or  job  redesign.  For  example,  the  job  redesign 
action  module  would  combine  several  guidelines  from  the  Organiza- 
tional Psychology  and  Engineering  Psychology  sections.  Based  on 
these  guidelines,  specific  actions  that  should  be  considered  in 
redesigning  a job  would  be  stated.  Examples  of  actions  might  be: 

(1)  analyze  worker  abilities  and  interests;  (2)  determine  function 
or  functions  to  be  performed,  (3)  identify  tasks  needed  to  perform 
required  functions,  (4)  obtain  worker  suggestions  concerning  group- 
ing of  tasks  into  positions,  (5)  form  groups  of  compatible  tasks 
having  sufficient  variety  to  maintain  worker  interest,  (6)  evaluate 
worker  reactions  to  redesigned  job. 

The  next  step  in  the  research  program  involves  development  of 
a tracing  system  for  linking  the  action  modules  to  the  diagnostic 
procedure.  There  are  a number  of  possible  approaches  to  this  pro- 
blem. One  procedure  that  seems  promising  involves  ratings  by  ex- 
perts of  the  relevance  of  each  action  module  for  each  of  the  items 
in  the  diagnostic  inventory.  This  procedure  would  yield  a vector 
of  weights  for  each  diagnostic  item  corresponding  to  the  action 
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module.  Once  an  organizational  profile  is  obtained  on  the  diag- 
nostic inventory  the  item  ratings  would  be  combined  with  the  action 
module  v;eights  and  summed  across  diagnostic  items.  This  would  pro- 
duce a score  for  each  action  module,  making  it  possible  to  rank  the 
modules  in  terms  of  their  relevance  for  the  organization  analyzed. 
Depending  on  a consideration  of  cost-factors  the  relevance  of 
various  action  strategies,  and  Che  interrelationships  among  Che 
action  modules,  decisions  can  be  made  with  regard  to  the  modules 
to  be  combined  to  form  a program. 

Evaluation 

Once  the  action  program  is  implemented,  the  results  will  be 
evaluated  in  terms  of  cost  and  behavioral  criteria  related  to  stated 
organizational  goals.  Initially,  this  will  be  done  in  an  effort 

to  validate  the  methodology.  However,  evaluation  is  essential  also 
due  to  the  fact  that  organizations  making  or  contemplating  changes 
must  conduct  careful  evaluations  in  order  to  derive  cost-effective- 
ness data.  Ultimately  the  method  will  be  successful  only  if  it 
leads  to  cost-effective  organizational  change. 

Concluding  Statement 

The  methodology  described  represents  an  ambitious  undertaking 
which  is  now  in  its  preliminary  stages.  Although  the  approach  re- 
mains to  be  validated  in  organizational  settings,  it  offers  a number 
of  advantages  over  conventional  organizational  development  techniques. 
The  method  is  explicit,  it  is  systematic,  and  it  offers  some  degree 
of  rigor.  Ultimately  it  should  be  possible  to  provide  the  capability 
for  organizations  to  diagnose  themselves  and  to  make  changes  to  meet 
human  rocourco  challenges  without  the  intervention  of  an  outside  party. 
This  is  the  ultimate  goal  of  the  procedure. 
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A.  Organizatio  icil  Objectives  and  Policies 

0-1  Extra  organ! zational  environment 
0-2  Organizational  objectives 
0-3  Organizational  decision  making 
0-4  Organizational  communication 
0-5  Attitudes  toward  safety 
0-6  Reward  policy 

B.  Job/Individual  Compatibility 

0-7  Job  design 
0-8  Worker  expectations 
0-9  Job  assignment 
0-10  Job  stress 

C.  Group  and  Interpersonal  Processes 

0-11  Establishing  group  norms 
0-12  Maintaining  group  norms 
0-13  Intergroup  competition 
0-14  Intergroup  co-operation 

D.  Leadership/Supervision 

0-15  Sources  of  influence 

0-16  Leadership  style 

0-17  Impact  on  safety  motivation 

E.  Organizational/Individual  Feedback 

0-18  Criterion  development 
0-19  Performance  counseling 


Outline  of  Behavior  Modification  Guidelines 

A.  Consequences  of  Behavior 

B-1  Reinforcement 
B-2  Punishment 
B-3  Avoidance 
B-4  Response  cost 

B-5  Social  reinforcement  - supervisory 

B-6  Modeling 

B-7  Peer  pressure 
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B.  Scheduling  of  Consequences 

B-8  Immediacy 
B-9  Consistency 
B-10  Shaping 

B-11  Partial  reinforcement 
B-12  Extinction 

B-13  Compensatory  reinforcement 
B-14  Stimulus  control 

C.  Pitfalls  in  scheduling  consequences 

B-15  Non-contingent  reinforcement 
B-16  Non-contingent  punishment 
B-17  Reinforcement  opportunity 

D.  Implementation  - General  Considerations 

B-18  Practice 
B-19  Record  keeping 


Outline  for  Training  Guidelines 

A.  Transfer  of  Training 

T-1  Training  and  testing  similarity 

T-2  Transfer  and  understanding 

T-3  Transfer  and  practice  on  the  original  task 

B.  Motivation  and  Trainee  Behavior 

T-4  Motivation 
T-5  Feedback 

C.  Conditions  of  Practice 

T-6  Guidance  toward  correct  responses 

T-7  Opportunity  for  correct  responses 

T-8  Mental  practice 

T-9  Learning  to  learn 

T-10  Massed-spaced  practice 

T-11  Part-whole  task  practice 

T-12  Stages  of  training 

T-13  Verbal  labelling 

T-34  Use  of  learning  time 

D.  Special  Considerations  Related  to  Training 

T-15  Overlearning  and  emergencies 
T-16  Learning  and  individual  differences 
T-17  Membership  in  the  work  group 
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Outlines  For  Engineering  Psychology  Guidelines 


A.  VJorK  Organization 


E-1  Function  allocation 
E-2  Shift  work 
E-3  Job  safety  design 
E-4  Work/rest  cycle 


B.  Job  Characteristics  and  Demands 


E-5  Arousal  and  stress 
E-6  Fatigue 
E-7  Work  pace 


C.  Workplace  Design  and  Layout 


E-8  Workplace  illumination 
E-9  Anthropometric  design 
E-10  Warning  signals 

E-11  Equipment  and  control-display  arrangement 

E-12  Location  of  visual  displays 

E-13  Location  and  protection  of  control  devices 


D.  Task  Characteristics  and  Demands 


E-14  Stimulus  response  (S-R)  uncertainty 
E-15  Task  information  processing  load 
E-16  Skilled  motor  performance 
E-17  Redundant  information 
E-18  Task  performance/memory  aids 


E.  Control-Display  Characteristics 


E-19  Stimulus-response  (S-R)  compatibility 
(population  stereotypes) 

E-20  Coding  and  identification 
E-21  Display  design 
E-2 2 Control  design 
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INDIVIDUAL  OUTCOMES  U ^ ORGANIZATIONAL  OUTCOMES 


[ORGANIZATION  OBJECTIVES 


REVIEW  OF  AIR  FORCE  JOB  SATISFACTION 
AND  CAREER  DEVELOPMENT  RESEARCH 


R.  Bruce  Gould 

Occupational  Research  Division 
Air  Force  Human  Resources  Laboratory 


The  zero  draft  environment  with  its  reduced  aptitude  and  educational 
levels  of  airman  inputs,  strengths  and  budget  reductions,  increased  train- 
ing requirements  through  technological  advances,  and  resulting  emphasis  on 
personnel  utilization  and  retention  provide  an  operational  requirement  for 
a job  satisfaction  research  program. 

The  Air  Force  Human  Resources  Laboratory  program  of  job  satisfaction 
research  began  in  1971  as  was  announced  to  the  Military  Testing  Association 
Conference  that  year  (Gould  & Christal,  1971).  The  effort  is  an  outgrowth 
of  the  United  States  Air  Force  Occupational  Survey  Program,  which  provides 
data  obtained  at  the  performance  or  task  level.  The  task-level  data  are 
central  to  this  research  program  and  differentiate  it  from  the  theoretical 
research  which  dominates  the  psychological  literature. 

This  paper  will  present  the  ongoing  studies  conducted  by  the  Air  Force 
Human  Resources  Laboratory's  Occupational  Research  Division  in  support  of 
the  job  satisfaction  research  plan  which  has  evolved  during  the  two-year 
period.  The  long-range  goal  of  the  plan  is  retention  of  job  qualified 
personnel.  To  this  end,  the  basic  steps  of  the  plan  are  to:  (^)  define 
the  dimensions  of  job  satisfaction,  (M  measure  the  satisfaction  levels  of 
these  dimensions,  (£)  identify  problem  areas  which  have  the  greatest 
potential  for  improvement  through  satisfaction  research,  (^)  measure  the 
effects  which  specific  changes  in  job  content  have  on  job  attitudes,  and 
implement  job  reengineering  actions  and  measure  the  effects  on  job 
attitudes  and  on  the  final  criterion,  career  decisions.  Although  the  steps 
in  the  long  range  program  are  logically  sequential  in  nature,  current  pro- 
jects studying  segments  of  each  step  are  under  way  to  provide  the  needed 
technology  for  longitudinal  studies  required  to  support  the  overall  objec- 
tive. An  overview  of  the  studies'  purposes  and  preliminary  implications 
will  be  discussed,  although  some  of  the  projects  are  still  in  the  process 
of  data  collection  or  analysis. 

Review  of  the  Literature 

To  take  advantage  of  previous  research,  some  400  civilian  and  284 
military  publications  have  been  reviewed  and  the  results  of  this  massive 
undertaking  are  in  press  (Tuttle  & Hazel,  1973).  To  summarize  the  find- 
ings: (a)  Job  satisfaction  appears  definitely  related  to  tenure,  absen- 

teeism, and  sick  calls,  however,  findings  investigating  the  relationship 
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between  satisfaction  and  performance  are  Inconclusive ; Job  satisfac- 

Lion  is  multidimensional,  with  satisfaction  and  dissatisfaction  represen- 
ting polar  ends  of  a continuous  attribute  rather  than  two  separate  attri- 
butes as  presented  by  Herzberg  (1966)  and  his  associates  (Whitsett  & 
Winslow,  1967) ; (c)  Job  satisfaction  is  a function  of  the  interaction 
between  work  factors  and  personnel  needs,  that  is,  the  extent  to  which 
expectations  or  needs  of  the  individual  are  fulfilled  by  his  work  environ- 
ment; (^)  Most  studies  have  dealt  with  subjective  reports  of  attitudes 
toward  jobs  — very  few  have  dealt  with  specific  job  content;  (e)  There 
have  been  few  dynamic  studies  relating  changes  in  job  content  to  changes 
in  attitudes;  (_f)  The  greatest  volume  of  literature  deals  with  overlapping, 
frequently  conflicting,  theories  of  job  satisfaction  which  are  probably 
all  oversimplifications  of  the  relationships  with  which  applied  researchers 
must  concern  themselves. 

Past  studies  tend  to  support  the  view  that  more  practical  applications 
of  satisfaction  research  will  result  from  studying  individuals'  satisfac- 
tion with  specific  facets  of  jobs  rather  than  using  a global  measure  of 
job  satisfaction.  Although  there  are  several  instruments  designed  to 
measure  satisfaction  with  specific  aspects  of  jobs,  notably  the  Minnesota 
Satisfaction  Questionnaire  (Weiss,  Dawis,  England,  & Loftquist,  1967)  and 
the  Job  Description  Index  (Smith,  Kendall,  & Hulin,  1969),  none  were  found 
which  covered  all  major  aspects  of  military  work  environments.  This 
necessitated  development  of  a suitable  Instrument,  applicable  to  all 
Air  Force  specialties. 

Development  of  the  USAF  Occupational  Attitude  Inventory 

Based  on  the  literature  review,  a list  of  presumed  attitude  content 
areas  or  facets  of  satisfaction  was  compiled  and  each  facet  briefly 
defined  and  reviewed  by  behavioral  scientists  familiar  with  the  military 
work  environment.  Thirty-four  work  dimensions  (shown  in  Table  1)  resulted, 
as  compared  to  20  in  the  Minnesota  Satisfaction  Questionnaire  and  5 in 
the  Job  Description  Index.  Items  were  then  written  by  three  behavioral 
scientists  for  each  defined  facet  resulting  in  an  original  pool  of  440 
items.  Using  6 independent  judges,  the  items  were  sorted  into  the  34 

dimensions  and  reviewed  for  clarity  and  redundancy.  For  selection  of  the 

final  items,  those  showing  classification  disagreement  were  inspected  for 
ambiguity  and  revised  if  possible.  Obviously  redundant  items  and  remain- 
ing ambiguous  items  were  removed,  resulting  in  a final  pool  of  348  items. 

Table  2 shows  an  example  of  five  of  the  specific  items  included  in  the 

inventory.  Full  documentation  of  the  development  is  nearing  completion. 

It  suggested  that  the  extensive  item  pool  and  literature  review  will  be 
useful  to  other  military  services  in  developing  their  own  service- 
specific  job  attitude  instruments. 


TABLE  1.  WORK  DIMENSIONS  OF  THE  OCCUPATIONAL  ATTITUDE  INVENTORY 


Achievement 

Personal  Growth 
and  Development 

Service  to  Others 

Activity 

Job  Design 

Social  Status 

Air  Force  and 
unit  policies 
and  practices 

Optional  Social 
Contact 

Sufficiency  of 
Training 

Assignment  locality 

Pay  and  Benefits 

Supervision  Received  - 
Human  Relations 

Authority 

Physical  Work 

Supervision  Received  - 
Technical 

Co-Workers 

Recognition 

Performance  Evaluation 

Creativity 

Required  Social 
Contact 

Job  Change 

Importance 

Responsibility 

Tools,  Equipment,  and 
Supplies 

Independence 


Physical  Safety  Utilization 


Knowledge  of  Results  Economic  Security  Variety 

Work  Schedule 

The  3A8  items  make  operational  use  of  the  inventory  prohibitive, 
because  of  examinee  attention  span  and  administration  time  constraints, 
but  they  provide  a useful  experimental  instrument.  The  experimental 
Occupational  Attitude  Inventory  has  been  administered  to  a random  sample 
of  5,800  airmen  and  the  returns  are  currently  being  analyzed.  An  8-point 
rating  scale  with  point  descriptors  ranging  from  Extremely  Dissatisfied 
to  Extremely  Satisfied  was  selected  for  inventory  use.  The  primary  goal 
of  this  analysis  is  to  reduce  the  instrument  to  the  smallest  subset  of 
items  which  will  measure  the  attitude  domain  variance  captured  by  all 
348  items.  To  do  this,  a computer  program  titled  VARSEL,  Variable 
Selection  in  the  Absence  of  an  External  Criterion,  has  been  developed. 
Using  an  accretion  method,  the  program  starts  with  one  item  and 
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iteratively  selects  items  having  the  greatest  unique  variance  of  all  the 
remaining  nonselected  items  until  the  combined  pool  of  selected  items  can 
predict  the  variance  measured  by  each  nonselected  item.  The  computational 
method  uses  a multiple  linear  regression  technique  and  has  options  for 
establishing  cutoff  points.  These  options  consider  item  reliabilities  and 
permit  investigator  Interaction  with  the  item  selection  process.  Prelim- 
inary findings  from  the  analyses  suggest  that  approximately  150  to  170 
items  will  remain  in  the  final  inventory,  and  that  these  items  measure 
about  40  internally  consistent  job  attitude  dimensions.  These  observations 
are  supported  by  profile,  regression,  and  factor  analyses. 

TABLE  2.  EXAMPLES  OF  SPECIFIC  JOB  SATISFACTION 
ITEMS  RELATING  TO  SUPERVISION 


1 = Extremely  Dissatisfied 

2 = Very  Dissatisfied 

3 = Moderately  Dissatisfied 

4 = Slightly  Dissatisfied 


5 = Slightly  Satisfied 

6 = Moderately  Satisfied 

7 = Very  Satisfied 

8 = Extremely  Satisfied 


Write  your 
answer  in 
this  column 


1.  The  extent  to  which  supervisor  keeps  you  informed  1. 


2.  Supervisor  fairness  in  rating  subordinates  2. 

3.  Way  supervisor  handles  teclmlcal  problems  3. 

4.  Supervisor's  knowledge  of  how  your  job  is  done  4. 

5.  Willingness  to  listen  to  employees  5. 


Development  of  an  Assignment  Preference  Survey 

An  additional  measurement  instrument  is  being  developed  to  evaluate 
attitudes  toward  assignment  locations.  Job  satisfaction  studies  in 
specific  specialties  indicated  that  assignment  location  frequently  plays 
a substantial  role  in  attitudes  towards  jobs.  The  current  assignment 
system  only  considers  positive  assignment  preferences.  Apparently,  these 
preference  ratings  are  often  made  by  individuals  who  are  relatively  ignorant 
of  the  physical  and  environmental  characteristics  of  the  locations.  The 
question  is,  what  are  the  environmental  characteristics  that  an  individual 
would  consider  in  selecting  assignment  preferences  if  they  were  available 
to  him?  Once  these  are  determined,  it  should  be  possible  to  develop  an 
assignment  system  which  provides  objective  Information  on  these  character- 
istics for  consideration  in  assigning  preferences.  Such  an  assignment 
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system  could  be  expanded  to  consider  not  only  a man's  likes  but  his  dis- 
likes. It  may  be  more  important  to  the  individual's  attitude  to  not 
assign  him  to  an  area  of  extreme  dislike  than  for  him  to  get  his  most 
preferred  assignment. 

An  Assignment  Location  Preference  Survey  has  been  administered  to 
8,000  airmen  to  provide  two  ratings: 

1.  Their  assignment  preferences  for  150  unnamed  CONUS  locations 
given  objective  data  on  factors  shown  in  Table  3. 

2.  Ratings  of  150  corresponding  base  names. 

Because  the  locations  correspond  to  the  bases  rated,  the  extent  of  base 
environmental  ignorance  among  airmen  can  be  evaluated  as  well  as  the 
environmental  factors  most  relevant  to  assignment  preferences. 

TABLE  3.  AIRMEN  - BASE  ASSIGNMENT  PREFERENCE  SURVEY 
FOR  150  MAJOR  CONUS  INSTALLATIONS 


ENVIRONMENTAL  FACTORS 

Base  military  population 
Size  of  community  nearest  base 
Miles  to  200,000  size  city 

Cost  of  living  compared  to  national  average 
Availability  educational  facility  (college) 
Distance:  mountains,  desert,  ocean,  lakes,  rivers 

Average  July  - January  temperature 
Average  summer  - winter  weather 
(humidity  - rainfall  - sunshine) 

Annual  snowfall 


Identification  of  Specialties  for  Indepth  Study 

Responses  of  some  175,000  airmen  job  incumbents  to  occupational  sur- 
veys in  150  career  ladders  are  available  for  identification  of  specialties 
for  indepth  study  of  job  satisfaction.  The  data  consist  of  background 
information  on  job  incumbents,  task-level  definitions  of  jobs,  and  responses 
to  a career  intent  statement  and  two  job  attitude  questions;  i.e.,  how 
interesting  the  job  is  and  how  well  it  makes  use  of  the  respondent's  talents 
and  training. 
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From  the  inventory  data,  we  can  state  that  most  airmen  find  their 
jobs  interesting  and  report  their  talents  and  training  are  well  utilized. 
However,  there  are  extensive  differences  within  and  between  career  ladders 
and  there  is  some  indication  that  over  the  last  few  years,  airman  reports 
of  dissatisfaction  have  been  increasing.  For  example,  in  the  150  career 
ladders  surveyed  to  date,  13  have  been  surveyed  twice  with  3-year  intervals 
between  surveys.  During  the  1967  to  1969  period,  24%  of  the  airmen  at  the 
3-(semi-skilled)  and  23%  at  the  5-( journeyman)  level  reported  their  talents 
and  training  as  utilized  "Very  Little"  or  "Not  at  All."  During  the  resurvey 
period  of  1970  to  1972,  42%  and  37%  of  the  3-  and  5-level  airmen  in  these 
ladders  reported  poor  utilization.  The  substantial  increases  in  reported 
dissatisfaction  may  be  due  to  any  of  several  reasons  as  in  deterioration 
of  jobs  or  restrictions  in  job  progression.  It  may  just  be  that  airmen 
are  more  willing  to  express  negative  attitudes  or  this  may  be  just  sympto- 
matic of  the  general  attitude  of  the  younger  civilian  population  from  which 
our  airmen  are  obtained.  Popular  civilian  mediums  are  reporting  substantial 
increases  in  dissatisfaction  with  work  performed.  Job  enrichment  programs 
are  becoming  the  vogue  in  civilian  industry  to  counter  the  trend. 

The  job  attitude  differences  among  career  ladders  can  be  illustrated 
by  comparing  the  percentage  of  airmen  reporting  that  their  talents  and 
training  are  poorly  utilized.  Rank  ordering  the  ladders  from  highest  to 
lowest  percentage  of  3-level  airmen  reporting  poor  utilization,  and 
comparing  the  two  extremes  of  that  distribution  (see  Table  4),  the  range 
is  from  63%  of  the  3-level  Pavements  Maintenance  personnel  to  none  of  the 
Dental  Laboratory  Technicians  (Gould,  1972).  Within  ladders,  the  percent- 
age generally  decreases  as  the  skill  level  increases,  especially  from  the 
5-  to  7-level.  Airmen  at  the  7-level  are  fully  qualified  E-6  or  E-7 
career  airmen. 

Comparing  the  specialties  at  the  extreme  ends  of  the  distributions, 
there  are  no  readily  apparent  explanations  for  the  differing  utilization 
reports.  At  both  ends  there  are  ladders  with  high  and  low  aptitude 
entrance  requirements  and  diverse  content  areas,  i.e.,  electronics,  equip- 
ment repair,  security,  and  medical  ladders.  This  suggests  that  there  are 
few  universal  causes  of  reports  of  dissatisfaction.  Individual  studies  of 
ladders  indicate  that  factors  related  to  dissatisfaction  are  essentially 
unique  to  each  career  ladder. 

Several  examples  will  be  cited  which  Illustrate  that  specific  problems 
are  unique  to  each  ladder  and  ladders  must  be  separately  studied.  For 
example,  in  the  Intelligence  Operations  ladder,  it  was  found  that  over  half 
of  the  tasks  airmen  were  trained  to  perform  in  the  mandatory  30-week  entry- 
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level  technical  schools  were  never  performed  in  the  field.  The  technical 
school  had  combined  the  curriculum  for  Photo  Interpreters  and  Intelligence 
Operators  because  of  common  elements.  This  had  the  dual  effect  of  increas- 
ing training  of  nonrelevant  tasks  and  increasing  the  required  aptitude 
entry  level.  Aptitude  requirements  are  set  by  predictability  of  technical 
school  success  and  the  photo  interpreter  tasks  were  highly  demanding  thus 
raising  the  aptitude  required  to  master  the  tasks.  The  curriculum  has 
been  adjusted  and  the  aptitude  requirement  lowered  to  the  requirements  of 
the  actual  tasks  performed. 

TABLE  4.  PERCENT  CLAIMING  TALENTS  AND  TRAINING  ARE 
UTILIZED  "VERY  LITTLE"  OR  "NOT  AT  ALL" 


AFSC 

Career  Ladder 

N 

3 

Skill  Level 
5 7 

551X0 

Pavements  Maintenance 

1059 

63 

54 

15 

276X0 

A/C  Control  and  Warning  Tech 

1689 

63 

51 

27 

461X0 

Munitions  Maintenance 

1590 

58 

50 

21 

402X0 

Aerospace  Photo  Sys 

261 

- 

56 

35 

671X3 

Accounting  Disbursement 

1354 

58 

45 

N/A 

611X0 

Supply  Services 

949 

53 

36 

12 

303X3 

Auto  Tracking  Radar  Rpmn 

708 

51 

39 

20 

234X0 

Precision  Photoprocessing 

836 

47 

42 

15 

811X0 

Security 

3617 

46 

36 

9 

443X0G 

Mlnuteman  Missile  Rpmn 

1032 

44 

46 

18 

204X0 

Intelligence  Operations 

926 

43 

39 

20 

915X0 

Medical  Materiel 

1022 

43 

26 

7 

811X0A 

Dog  Handler 

559 

10 

24 

2 

903X0 

Radiology 

548 

9 

12 

11 

432X0 

Jet  Engine  Mechanic 

1622 

9 

8 

6 

305X3 

Elect  Computer  Rpmn 

1055 

7 

14 

13 

607X0 

Loadmaster 

1773 

6 

12 

5 

324X0 

Precision  Measuring  Equip 

1212 

6 

7 

8 

322X1 

Weapons  Control  Sys 

1285 

5 

6 

3 

272X0 

Air  Traffic  Control 

1518 

2 

5 

7 

982X0 

Dental  Laboratory 

486 

0 

6 

3 
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In  the  Automatic  Tracking  Radar  Repairmen  ladder  it  was  found  that 
the  37-week  technical  training  concentrated  on  teaching  highly  sophisticated 
electronic  tasks  which  were  not  likely  to  be  performed  by  an  airman  until 
his  second  enlistment.  There  are  two  approaches  to  this  problem:  (£>  re- 

engineer the  Ist-term  airman's  jobs  to  coincide  more  with  those  of  the 
career  airman,  and  hence  the  initial  training  program;  or  (^)  provide 
initial  training  comparable  to  the  tasks  encountered  in  the  first  years, 
and  make  the  advanced  training  available  after  reenlistment. 

A different  situation  was  found  in  the  Accounting  Disbursement  ladder 
where  in  a single  year  (1968)  most  of  the  airmen  input  were  college  grad- 
uates who  scored  at  the  95th  percentile  on  all  aptitude  indices.  This 
career  field  has  many  demanding  jobs,  but  not  in  sufficient  proportions 
to  provide  challenging  jobs  to  such  an  Influx  of  talent.  The  problem  is 
not  likely  to  be  reencountered  now  that  we  are  experiencing  reduced  talent 
inputs  in  the  zero  draft  environment.  Another  issue  raised  in  the  study 
of  this  ladder  was  the  impact  which  supervisory  ability  has  on  job  atti- 
tudes. As  in  several  Air  Force  specialties,  this  ladder  is  combined  with 
other  ladders  in  the  accounting  specialty  at  the  7-level.  A study  now  in 
progress  is  using  occupational  survey  techniques  to  determine  critical 
supervisory  tasks  necessary  for  adequate  performance  as  a supervisor.  The 
results  will  have  implications  for  career  progression  patterns  and  career 
field  structures.  Interim  findings  in  other  ladders  indicate  that  adequate 
supervisors  are  important  to  Incumbent's  job  attitude  as  was  found  parti- 
cularly true  in  the  Aircraft  Control  and  Warning  Technician  ladder. 

To  date,  the  most  extensive  study  of  satisfaction  elements  in  a spe- 
cific ladder  was  conducted  in  the  Pavements  Maintenance  ladder  where  dis- 
satisfaction was  found  to  be  largely  a function  of  the  mundane  nature  of 
the  job  domain.  This  study  is  important  because  it  demonstrated  that 
occupational  survey  data  followed  by  field  interviews  can  be  used  to 
determine  the  nature  of  job  satisfaction  problems  and  provide  remedial 
recommendations.  However,  field  interviews  are  expensive  in  time  and  man- 
power and  the  next  question  is:  "Can  surveys  also  be  used  in  place  of  the 
field  interviews"?  Surveys  specially  designed  to  collect  the  data  pre- 
viously obtained  by  interviews  have  been  used  in  the  Munitions  Maintenance 
and  Aircraft  Control  and  Warning  ladders  and  are  currently  being  analyzed. 

One  of  the  more  ambitious  projects  is  the  development  of  a single 
large  data  file  for  the  study  of  airman  job  satisfaction.  Thus  far,  the 
occupational  survey  data  of  130,000  airmen  in  140  career  ladders  has  been 
matched  with  the  personnel  files  and  placed  in  a single  data  file.  The 
file  contains  task-level  performance,  job  attitude,  personal  background. 


VO  60  >0  60  90  100  10  20  X 40  60 

TAFMi  TAFMI 


One  of  the  first  uses  made  of  the  file  will  be  identification  of 
career  ladders  exhibiting  the  greatest  potential  return  for  Indepth  job 
satisfaction  study.  For  example.  Table  5 shows  regression  curves  for  four 
career  ladders.  Here  a job  attitude  dimension,  in  this  case  job  interest, 
is  represented  by  the  vertical  (Y)  axis,  and  months  of  Total  Active  Federal 
Military  Service  (TAFMS)  is  indicated  on  the  X axis.  In  predicting  inter- 
est, the  actual  predictors  used  are  career  status  (Ist-term  or  career 
airman),  tlrae-in-servlce  for  Ist-term  airmen,  service  time  for  career 
airmen,  aptitude,  and  the  TAFMS  values  squared.  Aptitude  is  included  in 
the  regression  model  for  several  reasons.  We  have  data  which  Indicate 
that  aptitude  levels  are  strongly  related  to  job  attitudes  and  these 
levels  vary  with  time.  Since  we  are  here  interpreting  cross-sectional 
data  in  a longitudinal  fashion,  the  effects  of  aptitude  differences  across 
time  must  be  held  constant. 

Compare  the  profile  of  the  Medical  Materiel  ladder  in  the  upper  left 
comer  of  Table  5 with  the  Operating  Room  Technicians  in  the  lower  right. 
Job  interest  decreases  for  the  Ist-term  Medical  Materiel  airmen  as  they 
progress  in  their  career  and  approach  the  reenlistment  decision  point  at 
the  48th  month.  The  jump  from  the  48th  to  the  49th  month  suggests  that 
those  reporting  uninteresting  jobs  did  not  reenlist.  The  magnitude  of 
the  jump  indicates  the  degree  of  relationship  between  job  interest  and 
the  reenlistment  decision.  Note  the  positive  acceleration  of  the  curve 
from  the  49th  to  the  100th  month  which  suggests  some  type  of  perceived 
job  progression.  A ladder  exhibiting  such  a profile  has  a high  likelihood 
of  impact  on  positive  job  attitudes  and  career  decisions  through  job 
reengineering  or  other  corrective  actions.  Such  actions  could  reverse  the 
trend  by  the  24th  month  and  produce  a positive  acceleration  in  job  attitude 
as  the  reenlistment  decision  point  is  approached. 

Ladders  with  profiles  such  as  the  Dental  Technicians  show  high  posi- 
tive job  interest  with  little  relationship  between  interest  and  reenllst- 
ment.  If  reenlistment  quotas  are  not  being  met  in  this  ladder,  the 
solution  is  probably  in  monetary  or  otlier  such  Incentives.  The  ladder 
shows  little  potential  return  for  effort  made  to  improve  the  jobs. 

Another  use  of  the  data  base  will  be  validation  of  statements  of 
career  intent  with  actual  reenlistment  decisions  made.  Extensive  regres- 
sion studies  will  be  performed  within  and  between  career  ladders  to  inves- 
tigate the  inter-relationships  of  job  factors,  background  and  aptitude 
factors,  and  job  attitudes  and  career  decisions. 


As  stated  earlier,  factors  related  to  dissatisfaction  reports  are 
essentially  unique  to  each  career  ladder.  There  are,  however,  a few 
relationships  which  have  been  consistently  found.  Those  job  Incumbents 
whose  jobs  are  more  demanding  and  varied  and  are  most  related  to  their 
aptitude,  education,  and  Air  Force  training  are  more  satisfied  with  their 
jobs  and  tend  to  express  positive  predispositions  toward  Air  Force  careers. 
Regression  studies  indicate  that  job  difficulty,  number  of  tasks  performed, 
and  aptitude  requirements  of  the  job  are  positively  related  to  job  attitude 
while  aptitude,  education,  and  TAFMS  for  Ist-term  airmen  are  negatively 
related. 

Studying  individual  career  ladders  leads  to  specific  restructuring 
of  jobs  and  suggested  changes  in  the  classification  system,  technical 
training,  and  aptitude  and  educational  requirements.  However,  the  process 
places  great  demands  on  limited  professional  resources.  General  guidelines 
and  methodology  will  assist  management  engineering  and  classification 
personnel  In  such  actions.  A potential  source  for  operationally  reviewing 
specific  job  structures  for  improvement  has  hardly  been  tapped;  i.e.,  the 
job  incumbents.  With  this  in  mind,  a pilot  study  has  sent  occupational 
surveys  to  personnel  in  the  Medical  Specialist  and  Ground  Radio  Communica- 
tion specialties.  The  surveys  request  each  job  incumbent  to  rate  all  tasks 
in  his  specialty  according  to  his  preference  for  performing  the  task.  A 
second  survey  requested  the  Medical  personnel  to  structure  "ideal  jobs." 

The  surveys  have  been  returned  and  will  be  subjected  to  cluster  analyses 
to  see  if  the  incumbents  can  provide  practical  clusters  of  tasks  that  form 
interesting  and  meaningful  jobs. 

In  summary,  the  Air  Force  is  two  years  into  its  program  of  job  satis- 
faction research.  Tlie  goal  is  to  sustain  an  all-volunteer  force  through 
making  maximum  use  of  available  talent.  The  operational  concern  is  to 
identify  the  dimensions  of  job  satisfaction,  to  measure  the  satisfaction 
levels  of  these  dimensions,  to  identify  problem  areas  which  have  the 
greatest  potential  for  Improvement,  to  measure  the  complex  interplay  of 
specific  job  content,  changes  in  job  content,  and  characteristics  of 
individuals  on  Job  attitudes,  and  finally,  to  reengineer  jobs  accordingly. 

It  is  a big  order,  and  the  program  has  limited  manpower  resources. 
Notwithstanding,  preliminary  studies  are  well  in  progress.  Current  pro- 
jects include  development  of  an  Occupational  Attitude  Inventory  and  an 
Assignment  Preference  Survey;  establishing  levels  of  job  satisfaction  for 
the  airmen  force  in  general  and  for  specific  career  ladders;  indepth 
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studies  of  specific  career  ladders  evidencing  high  levels  of  job  dissatis- 
faction to  identify  causes  and  suggest  remedial  actions;  development  and 
utilization  of  a massive  data  base  to  identify  potential  career  ladders 
for  detailed  study  and  to  provide  a medium  for  between  and  within  career 
ladder  studies.  Special  studies  are  being  conducted  to  identify  critical 
supervisory  experiences,  to  evaluate  the  ability  of  surveys  to  eliminate 
the  need  for  field  interviews,  and  to  permit  job  incumbents  to  have  input 
into  structuring  jobs. 

The  most  crucial  studies  are  in  the  design  and  planning  stages  — 
longitudinal  studies  which  will  relate  changes  in  job  content  to  changes 
in  job  attitudes  and  eventual  career  decisions. 
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A METHODOLOGY  FOR  DETERMINING  JOB  SATISFACTION 
IN  THE  U.  S.  NAVY 

lawrence  A.  Goldman 

Navy  Occupational  Task  Analysis  Program,  Washington,  D.  C. 


In  the  armed  services.  In  effect  an  All  Volunteer  Force,  It  Is  essential 
that  job  satisfaction  be  maximize!  while  job  dissatisfaction  be  minimized. 

There  are  three  major  reasons  for  doing  so.  First,  satisfied  individuals  are 
more  likely  to  make  the  military  a career.  Research  conducted  on  personnel  in 
private  industry  has  indicated  that  the  higher  the  overall  job  satisfaction, 
the  lower  the  turnover  rate.  Because  of  substantially  higher  personnel  costs 
needed  to  maintain  an  Al"'  Volunteer  Force,  it  Is  Important  to  keep  the  attrition 
rate  as  low  as  possible. 

Second,  knowledge  of  the  sources  of  satisfaction  and  dissatisfaction  will 
enable  the  military  to  more  effectively  utilize  its  lim.ited  available  personnel 
resources.  This  is  especially  true  re.'^rding  the  composition  of  the  personnel 
within  any  occupational  career  field  wnlch  may  be  affected  under  the  All  Volun- 
teer Force  concept.  For  exam.ple,  in  an  occupational  career  field  in  which  men 
predominate,  the  number  . f women  might  substantially  increase.  In  view  of  this 
possibility,  it  would  be  Important  to  determine  whether  or  not  women  are  satis- 
fied in  this  field,  in  order  to  help  decrease  their  rate  of  attrition. 

Third,  an  understanding  of  the  Sf urces  cf  satisfaction  and  dissatisfaction 
makes  it  possible  for  the  armed  services  to  improve  the  general  well-being  of 
its  personnel,  a major  concern  in  its  own  right. 

The  purpose  of  this  study  is  to  explain  a methodology  for  identifying  the 
major  sources  of  satisfaction  and  dissatisfaction  for  four  Navy  ratings  (occupa- 
tional career  fields  for  enlisted  personnel)  in  the  aviation  coimnunlty.  The 
focus  cf  this  study,  however,  is  upon  the  Aerographer ' s Mate  (AG)  rating,  an 
occupational  career  field  comprising  personnel  trained  as  weather  forecasters. 

For  the  AG  rating,  this  methodology  is  used  also  to  examine  the  relative  satis- 
faction between  groups  based  upon  military  paygrade,  sex,  mental  ability,  and 
reenllstraent  intention.  These  four  variables  were  selected  because  of  their 
potentially  important  bearing  on  the  success  of  an  All  Volunteer  Force. 

The  data  were  gathered  by  use  of  a job  satisfaction  questionnaire  contained 
within  the  task  inventory  booklet  for  each  rating.  Each  individual  was  required 
to  provide  responses  to  statements  associated  with  33  job  factors  which  emerged 
from  an  extensive  study  conducted  by  the  Canadian  Forces  Personnel  Applied  Re- 
search Unit.  For  the  AG  rating,  three  methods  of  administration  were  used  to 
collect  the  data  from  72?  Aerographer ' s Mates  at  82  different  activities  in 
paygrades  E-2  through  E-9,  representing  59'^  of  the  total  strength  of  this  rating: 
(a)  Administration  by  teams  of  occupational  analysts  from  the  Navy  Occupational 
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Task  Analysis  Program  (NOTAP) ; (b)  command  proctoring  by  an  individual  designa- 

ted by  the  commanding  officer  of  the  activity  surveyed;  and  (c_)  individual  mail- 
out  to  selected  individuals  throughout  the  fleet.  For  subsequent  data  collection 
efforts,  the  individual  mailout  method  was  discontinued  because  of  a poor  rate  of 
return. 

For  each  of  the  33  job  factors,  satisfaction  was  viewed  as  a discrepancy 
between  responses  to  a "How  much  is  there  now?"  item  and  a "How  much  should  there 
be?"  item.  Using  a five  point  Likert  scale  (ranging  from  minimum  with  a value  of 
"1"  to  maximum  with  a value  of  "5"),  the  discrepancy  for  each  participant  was 
determined.  Theoretically,  an  Individual  who  perceived  a particular  job  factor 
to  be  as  available  to  nlm  on  the  Jot  (How  much  is  there  now?)  as  he  would  like 
it  to  be  (How  much  should  there  be?)  would  rate  these  items  the  same.  A higher 
rating  on  "How  much  should  there  be?"  than  "How  much  is  there  now?"  indicated 
dissatisfaction.  Specifically,  the  magnitude  of  the  difference  between  the  two 
responses  on  each  Job  factor  was  viewed  as  a measure  of  the  extent  of  the  job 
factor's  deficiency.  An  individual  who  responded  ”1"  on  the  "How  much  is  there 
now?"  item  and  "1"  on  the  "How  much  should  there  be?”  item  would  be  just  as 
satisfied  as  someone  who  responded  "5"  and  "5"  respectively.  An  individual 
responding  "1"  and  "5"  respectively,  though,  would  be  very  dissatisfied  with 
that  particular  factor  since  the  difference  has  the  maximum  magnitude  of  U. 

Difference  or  discrepancy  scores  were  obtained  for  each  individual  on 
each  job  factor.  In  the  analysis  of  tne  AG  rating,  these  discrepancy  scores 
were  then  used  as  input  for  single  classification  analysis  of  variance  com- 
bined with  a compc-.slte  t test  as  described  by  Garrett  in  the  6th  edition  to 
his  Statistics  in  Psycholcgv  and  Education. 

In  the  event  that  the  analyst  only  has  access  to  the  number  of  individuals, 
means,  and  standard  deviations,  then  the  analyst  can  run  t tests  on  each  pair  of 
job  factors  sequentially,  the  type  of  _t  test  used  depending  upon  whether  or  not 
there  is  homogeneity  of  variance.  The  major  difference  between  these  two 
approaches  can  be  explained  as  follows.  Only  a single  standard  deviation,  the 
within  group  or  error  standard  deviation,  is  computed  for  the  composite  t test 
used  with  analysis  of  variance.  This  error  standard  deviation  is  based  upon 
all  scores  and  with  generally  high  error  degrees  of  freedom.  On  the  other  hand, 
as  many  standard  deviations  as  factors  (or  variable  categories)  must  be  com- 
puted whenever  separate  t tests  are  run.  According  to  Garrett,  the  within  group 
standard  deviation  furnishes  a better  or  more  reliable  measure  of  uncontrolled 
variation  than  could  be  obtained  from  separate  standard  deviations.  The  results 
obtained  from  both  approaches  are  essentially  the  same,  though,  whenever  the 
means  are  based  upcn  a substantially  large  number  of  cases.  Moreover,  whenever 
the  number  of  cases  is  large,  it  is  noted  that  it  essentially  makes  no  difference 
whether  or  not  there  is  homogeneity  of  variance.  The  other  three  ratings  in 
this  study,  the  Aircrew  Survival  Equlpmentman  (PR)  naming,  the  Photographic 
Intelllgenceman  (PT)  rating,  and  the  Aviation  Antisubmarine  Warfare  Operator  (AW) 
rating,  were  examined  by  the  use  of  separate  _t  tests  since  individual  case  data 
were  not  available  as  they  were  for  the  AG  rating. 

Regardless  of  the  type  of  _t  test  used,  statistically  significant  differences 
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(p<.05)  between  job  factors  and  between  groups  based  on  the  selected  variables 
were  then  determined.  Results  obtained  from  applying  either  approach  were  then 
used  to  place  the  job  factors  the  variable  categories  into  separate  clusters. 
Pertinent  to  tnese  clusters,  job  factors  or  variable  categories  appeared  ir;  two 
clusters  wnenever  they  were  statistically  equivalent  to  two  or  mere  other  fac- 
tors (or  categi,ries)  w’nlcn  were  statistically  diffei'ent  from  each  other. 

Table  1,  wiiich  is  found  on  pige  d,  iridicates  t.he  major  sources  of  satis- 
faction and  dissatisfaction  for  the  AG  rating.  Ten  clusters  of  these  33  job 
factors,  toge^-her  with  the  mean  discrepincy  for  each  factcr.  are  noted  in  •^his 
table.  In  the  AG  rating,  pay  and  allowances  (cluster  l)  was  the  most  dlssa’is- 
fying  aspec*  ..  p the  ieij,  ■while  aiequac;;  pf  living  condltl..ns,  recog::it xoi.  f r 
work  done,  and  prop_r  utlllzatici.  ^ incney  and/or  resources  (cluster  2)  were 
more  satisfying  than  pay  and  allowances  but  more  dissatisfying  t.han  the  c*  h.i-r 
factors.  Clusters  3 tiirough  10  can  be  si.milarly  interpreted.  It  is  observed 
that  freedom  from  frequent  job  changes  in  the  activity  'cluster  10)  was  the 
most  satisfying  job  factor,  perhaps  because  Aeregrapher ' s Mates  believed  that 
they  experienced  job  stability  to  a greater  extent  than  generally  exists  in 
otiier  Na\^  ratings. 

Table  2,  whicn  is  found  on  page  j,  summ'irizes  the  differences  wlrlo’t  "Xist 
between  military  poy grades  on  ^11  of  the  33  job  factors  for  the  AG  i-atii;g.  This 
table  indicates  that  w.henevei'  differences  occurred  between  pay.-trades,  ger--;-a 
E-6's  and  above  were  more  satisfied  than  E-5's  and  below  except  for  opp  r'* u;  ' ' v 
for  promotion  and  adequate  opo-ratlon  of  the  military  system.  A close  examiia- 
tion  of  tnls  table  reveals  that  wViile  E-3’s  and  E-9's  were  ti.e  most  satisfiei 
individuals,  E-'i's  were  the  most  dissatisfied.  In  order  of  increasing  iisca'i'- 
faction,  E-7's,  E-6's,  E-2’s,  E-3's,  and  E-A's  were  more  dissatisfiel  than  E-S's 
and  F-9's  but  more  satisfied  than  E-'j's. 

Analysis  of  tiie  job  factors  for  the  AG  rating  based  on  sex  revealed  t' 
women  were  as  satisfied  as  men  except  for  pay  and  allowances  in  which  wemen 
were  more  satisfied  than  men.  Despite  tills  one  difference,  men  and  vomer,  were 
equally  satisfied  since  one  or  two  significant  differences  could  occ-or  by 
chance  based  upi.n  33  sepirate  tests. 

Aiialysis  of  ixisic  mental  ability  is  based  upon  the  combined  General 
Classlficatl oil  Test  and  Arithmetic  Test  score  of  the  Navp ' s Basic  Test  Battery, 
in  which  these  six  ranges  of  scores  were  used:  (a)  less  tiian  100:  (h)  100-109; 

(c)  110-119:  (d)  120-129:  (e)  130-139:  and  (f)  l^i"©  and  i.ighev.  T!ie  composite 
General  Classi  f ioati-.  n Test  and  Arithmetic  Test  score  was  used  because  the  normal 
mini.mum  requirement  for  for.mal  sciiool  ^"A"  school)  training  in  the  AG  rating 
is  a ccmposlte  score  of  110  or  more.  Review  of  mental  ability  in  this  rating 
Indicated  no  significant  differences  between  tliese  six  categorizes,  except  that 
tiie  highly  intelligent  130-l-',9  itrcup  was  more  dissatisfied  than  the  other  iiiental 
ability  .p*'  ups  or  freedom  from  restrictive  rules  and  re  .-ail  at  lens,  worthwhileness 
of  work,  and  opportunity  for  }.elpir,.g  others.  Even  this  exception  is  not  con- 
clusive since  ‘■a.e  nighe.-y  men'al  ability  greup  ( lAo  and  tiigiier)  was  as  satisfied 
as  the  otticr  mental  ai  il'ty  groups  on  these  three  job  factors. 
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In  Mdiitiin  Lv,  the  bacK^rounl  vurlatles  of  military  paygrade,  sex^  and 
mental  al.iiity,  t.ne  reetil i stpaent  intentions  of  AG's  were  analyzed.  Concerning 
their  reei.i..!  ei,  -nc  . individual  responded  to  the  questici  "D’ 

You  Plan  to  RcenlistV"  by  eitiiei-  "Yes",  "No”,  or  "Uncertaiii" . As  sliown  in 
Table  3,  wi.icn  is  found  on  page  8,  there  were  significant  differences  for  20 
of  the  33  job  factors.  On  eight  factors,  individuals  planning  to  reenlist  and 
individuals  uncertain  about  reenlistment  were  more  satisfied  than  personnel  not 
planning  to  reeiilist.  On  ten  factors,  individuals  planning  to  reenlist  were 
more  satisfied  than  persoiuiel  uncertain  about  reenlistment  and  personnel  not 
planning  to  reeiiiist.  On  tiie  other  two  factors,  training  for  the  Job  and  gui- 
dance receiveu  on  the  ;ob,  individuals  planning  to  reenlist  were  more  dissatis- 
fied than  boti'i  personnel  uncertain  about  reenlistment  ar:d  personnel  r;ot  plan- 
ning tL.  reenilst.  Concerning  these  two  factors  for  which  those  planning  t' , 
reenlist,  bended  to  be  more  dissatisfied,  perhaps  additional  job  training  and 
guidance  slu.uli  be  pr...vided  AG's  as  inducement  for  a more  satisfying  careei'. 

The  raajoi-  purpose  of  the  methodology  used  in  this  study  was  tc  pr_vide  a 
better  assessment  of  Job  satisfaction  and  dissatisfaction  i.han  could  be  achieved 
through  rank  ordering.  Pertaining  to  Table  1,  pay  and  allowances  would  rank 
highest  if  rank  order  were  used  (that  is,  a rank  of  1 in  terms  of  greatest  dls- 
sati.sfacticn) . Adequacy  of  living  conditions  would  he  assigned  a rarlt  of  2, 
recognition  for  work  done  would  be  assigned  a rank  of  3>  etc.  A rank  ordering 
would  imply  that  a.iequacy  of  living  conditions  was  more  dissatisfying  ^han 
recogniti.n  f.  r work  dene,  etc.  Based  upon  statistical  criteria,  iiowever, 
there  were  usually  clusters  of  t'wo  or  more  Job  factors  tha^  were  eq\ially  sat- 
isfying (or  iissatisfying)  statistically. 

An  I'.qxtrtant  asjject  of  this  approach  is  that  reliable  results  would  be 
.ditainei  regariless  . f h.vv/  Job  satisfaction  was  defined.  Thiere  are  at  least 
nine  Isf"  i,  p.;.  ieflni  ■'"i  ons  of  Jtb  satisfaction  which  are  cit.ed  in 

tlk;  liter  i-nj-e.  iii' !u  i i r..g:  ^a'!  Tlie  "Is  Now"  model,  in  which  Job  sati  sfactl -.n 
i lefintfi  in  lArr.f.  ..f  resp. .rises  only  to  a "How  Mucti  is  There  Now?"  item  for 
e • j,  I 'i-o  !•:  ; (p)  liiiiiel.;  In  whicti  importance  ratings  are  used  to  weight 

varh  us  coml  ination.-  of  "How  'luch  is  Tiiere  Now?"  and  "How  Much  Should  Tlrere  Pe?" 
item..;  ( f'  r exn.iipbo,  iinpor  t-oijCe  X ^Should  Pe  - Is  Now)  , Importance  X Is  Now, 
and  imp  .r ta nce  - 1.;  Now'^ 


This  a,.proac,i  ..'un  to'  •pplimi  to  any  rating  or  combination  of  ratings.  In 
par;  icular . * he  res  1 1 1 .■  i fiined  from  analysis  of  the  AG  rating  can  be  CAornpired 
with  r.thez'  nav.' 1 r';  i.  i .i  o';  in  „ri-r  to  determine  the  Job  factors  which  are  generally 
indicative  'f  greatest  .satisfaction  and  dissatisfaction  in  the  Navy.  As  pre- 
viou.'ly  i nl  j ••  1 * e 1 , Mii  iias  i een  1 ne  already  for  three  other  ratings  - tire 
Aircrew  SirrvivaJ  P .u i p-a  iitm  u,  (PH)  rating,  the  Photographic  Tntelligenceman  (PT) 
rating,  and  the  Aviation  Anti subrniirlne  Warfare  (AW)  rating.  Tne  sources  of 
greatest  satisf.acti  .rn  'ind  dissatisfaction  for  these  tiiree  ratings  (together  with 
tnose  for  the  AG  rating)  ai-e  snewn  in  Tables  h and  5 on  pages  9 and  10  respect- 
ively. Tne  fact'  rs  wriich  represent  tiiese  major  sources  comprise  roughly  the 
highest  and  lowest  of  the  33  Job  factors  in  each  rating,  based  upon  either 
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the  first  three  clusters  of  factors  or  the  first  eight  factors  regardless  of  the 
number  of  clusters  in  which  these  factors  appear.  The  frequency  of  occurrence 
of  the  13  factors  representing  the  sources  of  greatest  satisfaction  and  the  I3 
factors  representing  the  sources  of  greatest  dissatisfaction  in  these  four  ra- 
tings is  shown  in  Table  6 found  on  page  11.  Review  of  Table  6 indicates  that 
while  individuals  are  given  the  opportunity  to  assume  responsibility  in  carry- 
ing out  their  assigned  duties,  and  while  they  get  along  well  with  their  co- 
workers,  they  are  dissatisfied  with  what  they  perceive  to  be  sufficient  recogni- 
tion for  their  efforts  as  well  as  with  their  overall  growth  and  development. 

Pay  and  allowances,  which  appears  as  a major  source  of  dissatisfaction  in  these 
four  ratings,  is  probably  more  satisfying  at  the  present  time  in  view  of  the 
recent  pay  increases.  It  is  noted  that  opportunity  for  promotion  appeared  as 
a major  source  of  dissatisfaction  in  three  of  these  four  ratings.  A substan- 
tially greater  number  of  ratings  should  be  analyzed,  however,  before  general 
trends  can  be  discerned  Navy-wide,  or  at  least  within  the  aviation  community. 

This  methodology  could  also  be  applied  to  occupational  career  fields  in 
the  other  branches  of  the  armed  services.  Through  the  use  of  this  methodology, 
recommendations  could  be  made  regarding  the  sources  of  job  dissatisfaction 
that  should  be  minimized  in  order  to  achieve  a more  effective  military  organiza- 
tion, especially  under  the  All  Volunteer  Force. 
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TABLE  1 


f 


Relat  i-vc  .'a  ‘ i tL  n of  Aeroctrapher ' s Mates  on  33  JoL  Dimensions 


Cluster 


Jot  Dimensions 


Mean  Discrepancy 


1 

2 

3 


h 


5 


6 

7 


8 


9 


10 


Pay  and  allowances 


2.0k 


Adequacy  of  living  conditions 
Recognition  for  work  done 

Proper  utilization  of  money  and/or  resources 

Opportunity  for  promotion 
Opportunity  for  growing  and  developing 
appeal 

Adequate  oiiei-ation  of  the  military  system 
OpportiUiity  for  achievement 

Opportunity  fci'  growing  and  developing 
Joi  apjitra.i 

Aiequate  pei-ation  of  the  military  system 

Opp'  r'.uiiity  f^  r acnievement 

Opp^.rtur. i rc/  t,  have  pj’ide  for  the  service 

Fair  treatment  ty  the  system 

A iequ-acy  . f work  surroundings 

Ti’alnir.g  f u-  tr.e  jol^ 

'Training  f.T  tiie  ,jct 
C' ..ripft.ence  ■ f 
J'd)  chalieng.r 
Resources  to  do  the  job 
.'pp'ii'tuf.ity  i'.ji’  pi’Gving  self 
.It'itu'  witl.in  tic'  organization 


fuatu.;  withii;  tir  i ttpitii za tion 
''h.  -.Pv'f;  ' . i>  t primary  ,iob 


Cli.-incf  t‘  1 . the  prirr.ai-y  job 
Cijmjx;  tf.nct.;  of  sube'i- iinates 
Acceptance  of  your  recommendations 
Opportunity  to  io  the  entire  ,iob 
V/orti;W:ii  lone:;s  ^..f  work 
Al-.;,uacy  i,f  relatiins  witu  superiors 
Guidaiice  received  on  ttie  job 
Opportunity  to  see  work  results 
Opportunity  to  contribute 

Freed.:."!  from  restrictive  rules  and  regulations 

f pI>ortuni  ty  to  -;ee  work  results 
Oppurtutiity  to  c.  iitribute 

Freedom  from  restrictive  rules  and  regulations 
Oppui-tur.ity  for  nelplng  others 

Respotisibili  t.y  and  independence  in  carrying  out  the  job 


Opportunity  f.,r  helping  others 

Respt)nsibility  and  independence  in  carrying  out  the  job 

T'Yeed  .m  from  job  pressures 

Alequac.,''  of  relations  with  subordinates 

Fr-=-ed  m fr'C’  frequent  job  chansres  in  the  activity 
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1.79 

1-75 

.51 

l.Ul 

:.l3 

1.L2 

l.)n 

l.lij 
l.ld 
; . h? 
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: .^4-0 

1 . ho 
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.0- 


l.OJ 


0.  • 

0.05 
i).  lU 


0.9"> 
O.Oi 
0 . Gh 
0.37 
0.3h 

O.S" 

D.8h 

0.78 

O.'Jn 


0.56 


n 


f 

j 


TABLE  2 


, t’or  which  Differences  Exist  between  Paygrales 


■ i L Di:r.eii;’1ons 

1'  . :i  T -M  I 

Oppiirtnui  1 f>”  irri. wing  and  developing 
I'ppii'c.ui  y'r.j  f.  r proving  self 
■r'r'i.n'^e  ‘ do  the  primary  ,ioh 

r pt  ’ :iC  '"Ct  i I’C  .'r  .1' 

W(.r*;,wo!  it'i.e:;.-  . ''  w.  I'k 
Ci-.i-,;  •'  ■ '’i  -u  with  rupericrs 

<1.  • ate 

:■  • • •.  j pi’.  . taerr 

■''ieari''.,  " ' i ■ n.'  witi:  ■iibor-i  i nates 

h' , ; t i.  ' tiia  .system 

Pi-.’  T. i;i  r ft:’  W'-i’K  done 
Oppi’t.xt.i  t.,,'  ‘ ree  w.-rk  results 

pp.  i ' l,  ’’i  r pr  'tioi. 

.■  i 1 th’.-  •■■j.litary  system 

?r  .;■»  :l,’..aticn  ca'  ’ncney  and/cr  resour- 

Opn  rtur  i';  ' =i‘’:iievement 


.1.  ■; 


•'  ii-iU’n-,,  r-i’  i I.  ■ w;:,  . . upcrl',.r.5 

i • [yf.  i-ai---.-  and 

••e.nriat.i . ... 

Reap.  ...ail  ii’’.,'  i ; iepe:  ience  in  c.arry- 
i ii  • a'  ; .(  ,a  ! 


Differences  which  Exist 

E-6's  and  above  were  more  satlsi'led 
than  E-5's  and  below 


E-8's  and  E-9's  were  more  satisfied 
than  E-7's  and  below 

E-U's  and  below  were  more  satisfied 
than  E-5's  and  above 

E-7's  and  above  were  more  satisfied 
than  E-6's  and  belcv;,  with  E-'  ' r> 
being  more  dissatisfied  th.an  tne 
other  paygrades 

F-'Vs  and  at.  .ve  we  ' .■  • .■a‘‘‘  ' " : 

than  E-a's  and  bel  w,  witii  K-  ' 

E-7, 's  i eing  more  .-’a"*  i s ‘’i-”  5 • ’ru:  - 

and  E-"'s 

E-4  's  and  E-5's  were  m- re  3 i -i,.  v i 
tiian  the  otlier  paygrad"" 

E-f'.s  and  above  were  r^:  ’’i  "'i'  : 

than  E-!*'s  and  beltw 

E-6's,  E-8's,  and  E-9'r  '^rere  moi’.: 
satisfied  than  t!ie  oti.er  pay  .-’rales 
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TABLE  3 


Job  Dimensions 
Job 


for  which  Differences  Exist  between  Reenlistraent  Intentions 
Dimensions  Differences  which  Exist 


Job  appeal 

Responsibility  and  independence  in  carry- 
ing out  the  job 
Opportunity  for  promotion 
Opportunity  for  proving  self 
Worthwhileness  of  work 
Job  challenge 

Fair  treatment  by  the  system 
Proper  utilization  of  money  and/or  resour- 
ces 

Recognition  for  work  done 

Freedom  from  restrictive  rules  and 

regulations 

Freedom  from  job  pressures 
Competence  of  seniors 
Adequacy  of  relations  with  superiors 
Opportunity  for  growing  and  developing 
Opportunity  to  have  pride  for  the  service 
Opportunity  to  contribute 
Opportunity  to  do  the  entire  job 
Opportunity  for  helping  others 

Training  for  the  job 
Guidance  received  on  the  job 


Individuals  planning  to  reenlist  and 
individuals  uncertain  about  reenllst- 
ment  were  more  satisfied  than  per- 
sonnel not  planning  to  reenlist 


Individuals  planning  to  reenlist  were 
mere  satisfied  than  personnel  uncer- 
tain about  reenlistment  and  personnel 
not  planning  to  reenlist 


Individuals  planning  to  reenlist  were 
more  dissatisfied  than  personnel  un- 
certain about  reenlistment  and  per- 
sonnel not  planning  to  reenlist 
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TABLE  h 


Tne  .Si  ui'c'ds  of  Greatest  Job  S.atlsfaction  in  the  AG,  PR,  PT,  and  AW  Ra*in<js 


AG  Rating 

' pportiinity  tr  see  work  results 
Cppoi’tuni ty  ti.  contribute 

Freeio.T.  fror.  restrictive  rules  and  regula- 
tion.; 

Opp< rtuni ‘y  f r helping  others 
Rerp -nsi!  : 1 i t,v  ‘in  i independence  in  carry- 
ing nt  ':.e  ;,..b 

Oppof-^uni  ty  f r nelping  others 

F.ecp.  1...- u 1 '.  1 1,/  and  iniepender.ee  in  carry- 

in.:  ii'  1 

Pv  ■ ■ ' b pressures 

Aleguai'y  f relations  with  subordinates 

Freedo.n  from  frequent  job  changes  in  ti;e 

a . • I i V i t V 


Fr  Rating 

Ci.:npe',ence  of  suboriinates 
[•1,1  r Mini  ty  see  work  results 
Free.i.  ni  ii’cnn  job  pressures 

f'l^or  :.ui  i ;.o  do  Uie  entire  job 

Aiecuncy  . 'i’  relations  witli  superiors 
. ' ' . . m frequen!-  job  changes  in  the 

■i.' I i vi  t _/ 

Hesp.n.ni  1 , i Ly  itii  in  iepeiidence  in  carry- 
ir:g  .'Ut  J ' b 

, ) V-  -y  . f nelatii  ns  with  supei'ior.s 

• i-  M . I'..,  71  r,- ..|u-;nt  jot.  changes  in  the 
■)c  tj  V ; ty 

F.-.'s  P' i 1 i I i ty  and  inlependence  in  carry- 
ing ut,  ' job 

>.  iei.u ''‘,7  ._i‘  relatl.ns  with  subordinates 


PR  Rating 

■ ppertunity  to  contribute 
Adequacy  cf  relations  with  superiors 
Wv.rthwhileness  of  work 

Adequacy  of  relations  with  superiors 
Worthwhileness  of  work 
Opportunity  to  see  work  results 

Opportunity  to  see  work  results 
Opportunity  for  helping  otlierc 

Opportunity  for  helping  other;; 

Adequacy  of  relations  with  subordinates 
Responsibility  and  independence  in  .‘arry 
ing  out  the  Job 


AW  bating 

Job  challenge 

Opportunity  to  contribute 

Adequacy  of  relatlon.t  with  superiors 

Re.'.ponsibi]  ity  and  in iepcn'lei  in  'a., 
ing  out  the  job 
'pportunity  for  helpin:?  o'her''- 
Freedom  from  frequent  jet  changes  in  Mb- 
n-  tivlty 

Adequacy  cf  relations  witii  subor.dii.ar.es 
Freedom  from  jol  pressure .t 
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TABLE  5 


The  Sources  of  Greatest  Job  Dissatisfaction  in  the  AG,  PR,  PT,  and  AW  Ratings 


AG  Rating 

Pay  and  allowances 

Adequacy  of  living  conditions 
Recognition  for  work  done 

Opportunity  for  promotion 
Opportunity  for  growing  and  developing 
Job  appeal 

Adequate  operation  of  the  military  system 
Opportunity  for  achievement 


PT  Rating 

Proper  utilization  of  money  and/or  resour- 
ces 

Pay  and  allowances 
Recognition  for  work  done 
Adequacy  of  living  conditions 
Opportunity  for  growing  and  developing 
Adequate  operation  of  the  military  system 
Status  within  the  organization 
Training  for  the  job 


PR  Rating 

Pay  and  allowances 
Adequacy  of  living  conditions 

Opportunity  for  growing  and  developing 
Job  appeal 

Adequate  operation  of  the  military  system 
Opportunity  for  achievement 

Recognition  for  work  done 

Opportunity  for  promotion 

Proper  utilization  of  money  and/or  resour 

ces 


AW  Rating 

Proper  utilization  of  money  and/or  resour 
ces 

Adequacy  of  living  conditions 

Recognition  for  work  done 

Adequate  operation  of  the  military  system 

Opportunity  for  promotion 

Opportunity  for  promotion 
Adequacy  of  work  surroundings 
Pay  and  allowances 
Fair  treatment  by  the  system 
Job  appeal 


TABLE  6 


. , , . u-i.  vT  the  Jcurces  of  Greatest  Job  Satisfaction  and  DlssHtisfacti  n 

in  the  AG,  PR,  PT,  and  AW  Ratings 

f ;r^ai--^st  Job  Satisfaction  Frequency  of  Occurrence 

i '-H  ■ i \y  ■-'tid  indepcudence  in  carry- 
ing V.UT,  t lie  I I: 

A ie.juacy  i i'  relari.ns  with  subordinates  ^ 


Oppor  *ur.i '.V  t o see  work  results  3 

Oppv-iruuii  !.  c^r;*ribut,e  3 

1 lip  .1-' nni  t ; t\n-  iielpLng  ottiers  _i 

Fi’eedim  I'ri  jot  pressures  3 

Free  i "<  !h-i  :;i  ft-equent  job  changes  in  tiie 

■iciLiviti'  3 

Adeiu'u-y  ■ f relations  with  superiors  3 


I'Vee  iv  ::i  ii\.m  restrictive  rules  at.d  regula- 

Wnr'.nwtiili- s werk 

Cottf-e' en-'-r  . t'  rui  riinates 

opf  ortun  i 'y  U ic  tlie  entire  job 

Jil  id.-  ' O Tig" 


r 'Greatest  Job  Dissatisfaction 

Frequency  o’  Occurrence 

’ j • • . < 

. .uaiK'.. 

- . 

'■  1 1 ■/ 1 rg  c.  !'  i i t.i  oi'.s 

a 

j..  ,,j-.  • .. 

I ■ 'no  -..'^itary  systera 

.'1 

'PP  *r  ‘ ut  it 

i'.  ••  pr  r,.  tier. 

3 

' pp  .r.ni. . 

t'  ;■  -r  wi’"!;  ievel.plng 

3 

J,  K ■ :.f. : 

|'•x*  tjt.  ■ 

li  ' n ,-(•  : ney  and,' or  rescur- 

Ceri 

■ I t.  !i''  .in  i 

,/  f ■■  acliievement 

. ’ * ■ t M w 1 * 

t ne  organization 

' 

■ r t,  e ,'i.  ! 

L 

Aie|in.-;/ 

•'  '/  rk  :-iirr  'indings 

1 

F':  i 

...M  ' ( tlie  system 
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Interest  Surveys  and  Job  Placement  in  the  Armed  Services 


With  the  demise  of  the  draft,  the  armed  services  no  longer 
have  a captive  pool  from  which  to  create  their  future 
soldiers,  sailors,  and  airmen.  Instead,  they  must  attract 
and  retain  recruits  in  competition  with  business  and  in- 
dustry. In  addition,  through  the  "up  or  out"  policy,  the 
armed  services  are  trying  to  upgrade  significantly  the 
quality  of  their  work  force,  thus  putting  themselves  even 
more  directly  in  competition  with  industry  for  the  ablest 
workers . 

The  military  seem  to  be  hampered  in  this  competition  by  a 
number  of  restraints--their  on- the-surf ace  wage  scales  are 
inflexible  and  generally  inferior  to  those  offered  by  in- 
dustry, their  recruits  cannot  often  be  employed  near  their 
own  homes,  their  training  is  more  rigorous  and  regulations 
more  restrictive  than  those  commonly  encountered  in  industry. 
The  armed  services  have  already  initiated  significant  changes 
in  all  these  areas,  but  none  that  seem  likely  to  overcome 
completely  the  competitive  disadvantage.  In  fact,  present 
levels  of  recruitment  are  notably  short  of  what  is  needed, 
and  the  proportion  of  Class  IV  recruits  is  dangerously 
high.  The  problem  from  the  point  of  view  of  the  recruiting 
officer  is  almost  one  of  a textbook  case  in  personnel  manage- 
ment: how  can  a firm  in  this  situation  maintain  an  adequate 

work  force  and  avoid  excessive  turnover? 
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The  textbook  solution  is  easy  to  state  as  a general  principle 
— it  is  to  emphasize,  and  even  create,  areas  of  competitive 
advantage.  Significant  efforts  along  these  lines  are  also 
in  evidence.  Billboards  proclaim  that  military  life  is  more 
interesting,  more  challenging,  than  other  ways  of  life. 
Enlistment  is  heralded  as  the  opportunity  to  learn  a trade. 

Even  the  necessity  to  live  away  from  home  may  be  seen  as 
an  inducement,  rather  than  a disadvantage.  The  purpose  of 
these  remarks  is  to  consider  interest  surveys  within  the 
context  of  a job  placement  program  that  may  offer  another 
competitive  advantage  over  the  alternatives  offered  to  the 
young  adult  by  business  and  educational  institutions. 

There  is  no  doubt  that  money  in  the  pocket  provides  a strong 
motivation  for  a workman  to  seek  a new  job  or  to  stick  to  his 
present  task.  But  it  is  not  the  only  motivation.  Entrepreneurs 
who  are  ready  to  turn  their  hands  to  whatever  activities 
offer  the  largest  profits  are  presumably  still  to  be  found  on 
the  staffs  of  large  conglomerates,  and  they  are  undoubtedly 
the  sine  qua  non  of  crime  syndicates,  but  they  are  by  no 
means  typical  of  the  young  adults  who  are  the  targets  of  pre- 
sent military  recruiting  efforts.  A pertinent  study  has  just 
recently  been  reported  in  the  Vocational  Guidance  Quarterly 
(1)  . A sample  of  Amy  and  Air  Force  personnel  stationed  in 
West  Berlin  were  asked  to  rate  a number  of  personal  and  economic 


needs  in  terms  of  their  importance  to  the  individual's  satis- 
faction with  his  job,  and  also  in  terms  of  the  degree  to  which 
these  needs  were  satisfied  by  the  individual's  current  military 
activities.  The  results  of  the  study  suggested  that  needs 
for  security  and  socio-economic  status  were  being  met  to  an 
even  greater  extent  than  they  were  felt  to  be  needed,  while 
needs  related  to  self-actualization — such  as  self-expression 
and  creativity — were  not  being  met  to  the  extent  tliey  were 
felt  to  be  needed.  In  short,  these  men  tended  to  feel,  though 
they  might  never  have  replied  so  to  a direct  question,  that 
they  did  not  need  more  money  as  much  as  they  needed  a task  better 
in  accord  with  their  own  sense  of  themselves.  This  particular 
study  encompassed  personnel  of  all  grades  from  junior  enlisted 
personnel  to  senior  officers.  Theie  are  many  indications 
that  among  young  people  the  same  feeling  is  even  more  prominent. 
Today's  youth  are  seeking  more  than  a high  salary.  Of  equal 
or  greater  importance  to  them  is  a match  between  their  interests 
and  the  challenge  presented  by  the  job. 

Let's  assume  for  the  moment  that  there  are,  somewhere  in  the 
nation,  a goodly  number  of  young  people  to  whom  life  in  the 
modern  armed  services  would  not  be  largely  uncongenial.  Let's 
suppose  that  most  of  them  are  mature  enough  to  know  that  no  job 
anywhere  will  satisfy  their  wildest  dreams  every  moment  of  the 
day.  And  let's  suppose  further  that  some  of  them  present  them- 
selves to  a recruiting  officer.  Possibly  at  present  he  has  no 
task  more  urgent  than  to  enlist  them  as  quickly  as  possible. 
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But  one  key  basis  for  a volunteer  army  is  not  just  filling 
available  positions,  but  filling  them  with  applicants  who 
can  do  the  job  well;  and  not  only  applicants  who  can  do  the 
job  well,  but  applicants  who  are  likely  to  want  to  continue 
to  do  the  job  when  their  initial  enlistment  period  runs  out. 

I am  sure  it  is  no  news  to  members  of  the  Military  Testing 
Association  that  identifying  such  applicants  means  taking 
interests  into  account.  In  fact,  in  my  preparatory  review 
of  the  literature  on  this  subject  I found  that  the  ground  had 
been  well  prepared  by  Allen  Schuh,  of  the  Naval  Aerospace 
Medical  Institute  in  Pensacola.  As  a result  of  a thorough 
survey  of  studies  published  up  to  1967  (2) , he  concluded 
that  interest  inventories  are  better  predictors  of  employee 
tenure  than  are  either  intelligence  or  aptitude  tests. 

Not  much  has  turned  up  since  then  to  challenge  that  conclusion. 

Of  course,  this  is  not  meant  to  disparage  the  use  of  aptitude 
or  ability  tests.  The  typical  pattern  in  industrial  em- 
ploym.ent,  not  surprisingly,  is  that  groups  of  employees  whose 
interest  and  abilities  both  match  their  jobs  will  ha^’e  the 
longest  average  tenure,  while  groups  of  employees  who  have  a 
mismatch  in  either  interests  or  abilities  will  have  about  the 
same  but  shorter  average  tenures.  And  employees  who  don't 
match  the  job  in  either  interest  or  ability  don't  last  at  all, 
(except  possibly  in  the  army  during  wartime) . 
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Neither  is  it  any  secret  that  taking  interests  into  account 
means  some  kind  of  systematic  interest  survey.  It  is  not 
enough  to  ask  a young  man  whether  or  not  he  would  like  to  be 
a movie  actor  or  a hogshead  hooper.  In  the  one  case  he  may 
be  thinking  of  John  Wayne  while  you  are  thinking  of  a wound 
victim  in  a training  film;  in  the  other  case  he  may  have  no 
idea  at  all  of  what  is  meant.  But  to  show  that  I am  well 
aware  I am  carrying  coals  to  Newcastle,  I might  mention  that 
the  subject  was  well  covered  in  1963  in  A Guide  to  Psychological 
Tests , by  Joseph  Zeidner,  who  was  then  chief  of  the  Support 
Systems  Research  Laboratory  of  the  U.S.  Army  Personnel  Office. 

As  he  put  it,  "it  has  been  well  demonstrated  that  interests 
cannot  always  be  validly  measured  by  direct  questions,  especially 
by  people  with  little  experience  or  relevant  information."  (3) 

I tend  to  disagree  with  Zeidner' s review  only  about  his  pre- 
ference for  manifest  and  tested  measures  of  interest.  A 
person's  manifest  interests  are  determined  by  things  he  had 
done — he  is  interested  in  carpentry  because  he  has  built  a boat, 
or  in  driving  because  he  has  a driver's  license.  A person's 
tested  interests  are  measured  by  his  acquaintance  with  the 
technical  vocabulary  of  various  fields — if  he  thinks  a bit 
belongs  in  a horse's  mouth,  you  make  him  a cavalryman  and  not  a 
computer  programmer.  But  the  same  problems  arise  with  these 
approaches  as  with  direct  questions;  few  of  today's  new  re- 
cruits will  have  the  opportunity  to  gain  experience  or  under- 
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standing  in  some  of  the  most  critical  areas  of  employment 
in  the  modern  armed  services,  such  as  radar  technology  or 
computer  programming,  for  example. 

The  alternative  to  looking  at  manifest  or  tested  interests 
is  to  use  an  interest  inventory,  such  as  the  Kuder  Occupational 
Interest  Survey  or  the  Strong  Vocational  Interest  Blank.  These 
are  probably  the  two  most  widely  used  of  such  instruments,  and 
since  SRA  has  a vested  interest  in  one  of  them,  the  Kuder  sur- 
vey, we  might  take  a moment  to  describe  some  of  their  differences. 
Strong's  questionnaire  asks  subjects  to  rate  various  activities 
on  a like-dislike  scale,  while  Kuder 's  asks  them  to  pick  the 
most  liked  and  least  liked  of  a set  of  three  activities.  Form 
C of  the  Kuder  produces  profiles  on  a set  of  ten  normed  scales 
with  titles  like  Outdoor,  Computational,  Persuasive,  Clerical, 
etc.  The  entire  profile  can  be  compared  with  average  profiles 
for  many  different  occupations.  Strong's  instrument  provides 
22  scales,  with  titles  directly  associated  with  occupational 
groupings,  as  well  as  separate  scales  for  different  occupations. 
Kuder 's  form  DD  gives  a numerical  index  for  the  degree  of 
match  between  the  subject's  responses  and  the  average  responses 
of  a criterion  group  in  different  occupations. 

But  the  general  nature  of  these  instruments  is  undoubtedly 
well  known  to  anyone  much  involved  in  military  testing.  A 
recent  listing  of  our  industrial  and  government  customers  for 
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Kuder  C and  Kuder  DD  shows  a Veteran's  Administration 
Hospital  on  nearly  every  other  page,  several  military  re- 
habilitation programs,  and  even  some  Army,  Navy,  and  Air 
Force  bases.  In  fact,  the  widespread  use  of  interest  sur- 
veys in  a program  of  careful  job  placement  may  constitute 
a competitive  advantage  for  the  armed  services  in  attracting 
and  retaining  personnel.  We  know  there  are  many  commercial 
firms  that  use  these  surveys,  but  we  also  know  there  are 
a great  many  more  that  do  not.  We  tend  to  suspect  that  the 
greatest  use  of  interest  surveys  is  made  in  placement  at  the 
management  level,  and  we  fear  that  hourly  workers  are  al- 
most forgotten.  There  is  probably  a tendency  in  industry 
to  hire  whomever  they  can  get  during  periods  of  manpower 
shortage,  and  to  hire  only  the  most  able  during  periods  of 
manpower  surplus — with  the  idea,  perhaps,  that  the  new  em- 
ployee is  darn  lucky  to  get  the  job,  and  will  remain  ever- 
lastingly grateful.  It  isn’t  always  so.  In  fact,  you  of  the 
Military  Testing  Association  might  take  pride  in  your  role 
in  inspiring  the  following  bit  of  advice,  in  a text  on  personnel 
management:  "A  young  18-to-20-year-old  high  school  or 

technical  school  graduate  will  be  the  skilled  machinist, 
experienced  planner,  valued  draftsman,  or  even  foreman  of  the 
1970 's  and  1980 's.  He  needs  as  careful  selection  as  the  college 
graduate,  but  he  rarely  gets  it  from  industry.  That  is  one 
reason  why  many  of  the  brighter  youngsters  with  potentials  in 
these  areas  are  looking  at  other  careers — primarily  in  the 
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armed  services  and  in  government."  (4)  If  this  does  not 
entirely  square  with  your  experience,  perhaps  some  new 
factors  have  entered  the  situation  since  1967,  when  this 
was  written. 

Nevertheless,  the  educational  and  training  opportunities 
in  the  armed  services  are  exceptional,  and  are  rightly  ad- 
vertised as  such.  No  industrial  concern  offers  such  a 
broad  range  of  careers,  and  no  educational  institution  has 
such  an  excellent  program  of  job  placement  for  its  trainees. 

I think  it  is  clear  that  interest  surveys  can  well  serve 
the  interests  of  recruits  to  such  a program. 

But  of  course  we  have  also  the  interests  of  the  services 
themselves  in  mind.  There  are  peculiarities  to  interest 
surveys  that  allow  them  to  be  used  for  purposes  for  which 
they  have  very  little  face  validity.  The  Kuder  Form  DD,  for 
instance,  asks  for  no  choices  which  would  seem  to  distinguish 
between  electrical  engineers  and  heating  and  air-conditioning 
engineers,  yet  in  the  process  of  averaging  patterns  of  re- 
sponses for  groups  of  people  in  these  two  professions,  we 
come  up  with  scales  that  do  a pretty  fair  job  of  distinguish- 
ing between  the  two  groups.  We  do  a better  job  distinguishing 
between  electrical  engineers  and  lawyers,  to  be  sure,  but 
some  apparently  close  distinctions  can  still  be  usefully  made. 
We  have  had  some  success  in  predicting  academic  performance 
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from  interest  survey  responses  alone.  And  there  are  several 
studies  of  their  usefulness  in  identifying  psychological 
abnormalities . 


All  of  this  is  by  way  of  suggesting  the  usefulness  of  interest 
surveys  in  identifying  recruits  who  are  not  only  able  to  do 
the  various  tasks  that  the  modern  armed  services  need  to 
have  done;  but  who  will  be  satisfied  to  do  them  under  the 
conditions  of  life  that  the  armed  services  must  impose.  Pro- 
vided that  there  are  computer  programming  specialists  who 
re-enlist  even  though  they  might  find  more  highly  paid  jobs 
elsewhere,  some  experience  with  their  interest  patterns 
is  likely  to  be  invaluable  in  selecting  the  new  recruits 
"most  likely  to  succeed"  in  terms  of  successful  job  per- 
formance over  a suitably  long  period  of  time. 


I 

I 

I 
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Job  Description  in  the  Armed  Forces  as  a Basis 
for  Analysis  and  Evalaatlon 

Flottillenaitiiurnl  Fleblg 
German  Navy 

Opinions,  as  we  all  know,  differ,  sometimes  widely,  on  the  possibilities 
and  value  of  describing,  analysing  and  evaluating  jobs  and  duty  positions. 

In  the  Federal  Republic  of  Germany  the  legal  situation,  under  the  "Act 
on  the  Constitution  of  F.nterprlses"  since  1972  had  been  that  private 
employers  are  required  to  provide  employees  with  concrete  Information 
concerning  positions,  in  particular  with  respect  to 

- the  nature  of  the  job, 

- its  place  in  the  organization, 

- the  duties  and  responsibilities  associated  with  it. 

Although  the  Act  does  not  elaborate  on  this  point,  one  of  its  effects  is 
bound  to  be  some  measure  of  systemization  of  job  descriptions  in  trade 
and  Industrv.  First  beginnings  in  that  direction  are  already  being  made. 

Thus,  the  Federal  IIlnistr>'  of  Labor  and  Social  Affairs,  for  Instance, 
promoted  and  helped  to  establish  an  institution  called  "Committee  on 
Man  in  the  Administration"  whose  aim  it  is  to  promote  the  cooperation 
of  experts  from  the  scientific  communitv,  industry,  the  Public  Service, 
the  Trade  Unions,  Employer  Associations,  and  the  Labor  Administration 
so  that  common  solutions  to  the  many  problems  of  quantitative  and 
qualitative  manpower  planning,  development  anil  administration  might  be 
developed  and  propagated,  using  modern  scientific  findings  and  technologies. 
In  addition,  a number  of  private  organizations,  such  as  the  "Academy  for 
Work  Science"  and  the  "IL\Y  Business  Consultants"  firm,  for  instance,  are 
engaged  in  rcsearcli,  teacliing,  and  ronsnltative  work  in  the  field  of 
structural  investigations,  work  analysis,  and  job  description.  Other 
organizations,  such  as  the  Geriman  Shell  AG,  for  instance,  have  developed 
their  own  techniques  and  procedures. 

Independent  thereof,  an  analogous  trend  for  the  entire  public  service 
sector  of  the  Federal  Republic  has  become  unmistakable  since,  in  1973, 
a commission  appointed  by  the  Federal  Government  to  study  the  reform  of 
public  service  law  submitted  concrete  proposals  on  how  tlie  public  career 
and  pay  system  sliould  be  objectivated  by  putting  it  on  a function-oriented 
footing.  One  of  the  measures  recommended  to  that  end  was  to  develop  a 
number  of  Instruments  which  should  first  be  tested  in  selected  areas. 

This  is  the  larger  framev/ork  in  which  the  efforts  and  work  of  the  Federal 
Ministry  of  Defense  mtist  be  seen  on  which  T will  now  report. 


in  the  spring  v'l  this  year  (jy’J)  we  b<  g.in  vit’.  t:!;.  testing  of  a set  of 
instrorjeiUH  for  surveying  and  reorganizing  the  jobs  In  the  German  Armed 
IGiices,  '•  u oth-.-t  uordtj,  a.iiuost  two  yesis  hy  tef-ri  the  central, 

basic  p'^Jpo^ial  contained  in  the  report  b\  th^  ;iar.i  .-v  e .•  Structure  Commission 
appoi.itei  bv  the  Ministry  of  Defense  - namely  to  develop  an  Armed  Forces  - 
spe.citic  sy.'ice.ro  of  Job  description,  Job  analysl'^’  and  job  evaluation  - had 
finally  been  realized  to  the  point  where  we  had  an  instrument  ready  to  be 
tested. 

In  Its  investigations  the  Manpower  Structure  Commission  found  that  the 
present  evaluation  and  pay  system  in  the  Federal  Armed  Forces  Is,  in  the 
final  analysis,  based  on  rank.  Rank,  however,  it  determined,  is  no  longer 
indicative  of  the  duties  incumbent  on  the  individual  holding  such  rank, 
for  one  and  the  same  rank  today  often  embraces  a number  of  highly  diverge;,. 
Jobs.  Concrete  individual  Jobs  are  hence  siibsuiueu  under  an  abstract  rank 
whicli  in  turn  determines  the  rate  of  pay. 

Past  nK:thods  of  Job  evaluation  are  neither  up  to  raodern  standards  nor  do 
they  fit  the-  specific  needs  of  modern  armed  forces.  The  content  of  a 
position,  t.e.,the  Job  and  the  training  and  qualifications  required  to 
fill  It,  are  not  sufficiently  taken  into  account,  considering  modern 
work  science  standards. 

Relatively  large  differences  in  the  degree  of  si  ini:  involved  In  other- 

wise Identical  Jobs  with  a like  rating,  for  ii..')taa.;i  , have  a considerable 
disturbing  effect.  Tliese  differences  are  completely  ignored  in  the  current 
evaluation  and  pay  system,  which,  again,  has  ill  effects  on  personnel. 

Where  unequal  Jobs  are  subsumed  under  the  same  rank,  the  differences  and 
disturbing  effects  are  even  greater,  of  course. 

I'u'eei  j.rog'.'c  ss  ion , in  !lie  sense  that  .r.i  individual  may  clirai;  on  the  pay 
nc.ile,  i -s  possible  in  ilic  bundeswehr  todnv  almost  only  by  way  ol  promotion 
to  a iii.glKT  ank.  Iho  .umsequeuce  i.s  a iary.-.  r ol  relatively  high 

rank.s  without  a correspondingly  broad  b.xsc  of  entrance  grades.  An  indlvidu 
luy  lu  proi'’otod  to  a higher  rank  ai  though  rhe  Job  ho  pel  forms  does  not 
'variant  ,i  Ijigln  r vati.i:,.  .Assignment  pl.ui.iia;  Ui,  t;.e  e'.aluatlon  and  pav 
system  in  ihe  Ci>rdi.'Swo:i  r are  insofar  con  ; . t-  I '!•  .-.crious  difflcuitiof 

v.’lrich  ,ro  i to  no  i i!)  , , i-  ’o'.;;,)'  a completely 

nev,'  in  ;.ii  i i enii‘,i\-or  it  woni.i  . , ..  : o apply  to  the 

nni-nuty  ; '.lit  j.ibs  in  tlie  A 'med  Fort.;-.  i.  ...  ■-.urvi-v  and  evaluation 
techntqii.  . i.s,;d  in  t.,  ‘ private  sector.  i.i..  .-t  ...i  ricr  die  advantage 

I htu  the  public  aiui  the  private  sector  './ouid  m • ■ os.',  together  In  such 
lie.ld;-  1 . ..;i ..  iiig,  Jot  eqnivalencv  and  me  o;  m'  .rchange  of  personae 

..s  not  01..V  :!io  e.'.'.j  uati  on  technique.s  but  a S'  i-.--  e;  npational  classiflca- 
t,fon  S..1I  ■•vo.^-.'t.d  for  industry  could  be  dr-.  •.  .j,.-..  hi  revamping  tlie 
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system  of  duty  positions  in  the  Armed  Forces.  A grouping- together  of 
military  jobs  by  more  functional  characteristics,  which  might  become 
necessary  in  the  process,  could  result  in  more  efficiency.  We  will  have 
to  examine  very  carefully  thougli  which  ]obs  are  so  peculiariy  military 
that  their  evaluation  requires  a departure  from  generally  applicable 
criteria. 

The  Manpower  Structure  Commission  was  guided  by  the  basic  Idea  underlying 
the  present  system,  namely  that  public  ser'/lce  law  was  originally  concerned 
with  discrete  jobs.  Tlie  ever  increasing  number  of  discrete  jobs,  however, 
today  not  only  requires  that  they  be  systematl caily  re-inventoried.  They 
must  also  be  ordered  In  larger  groups  and  weigiited  by  the  degree  of 
executing  and  directing  responsibllltv  they  involve  In  order  that  the  source 
of  public  serv'lce  law  may  again  become  transparent  and  manageable  In  its 
further  evolution.  To  that  end,  tlie  Commission  held,  a system  of  job 
description,  job  analysis,  and  job  evaluation  must  be  developed  which 
permits  a Jl  f f ercnt  i area  .-i..sessmeut  to  be  niaJi.  or  tlie  inaividual  jobs  in 
their  various  dimensions.  Jla.sed  on  sucii  a sy.-jtt'.n  a new  persoiutei  structure 
for  the  Bundeswehr  niust  be  devised  which  satisfies  the  needs  of  ntidern 
armed  forces.  Let  me  summarize  these  needs,  as  we  see  them,  In  five  main 
points  : 

1.  Ail  individual  jobs  required  to  cnahli'  th  ■ t-.rmed  F>.rces  to  fulfill 
their  mission  must  be  c.apable  of  being  nnu.r  Loiij  1 1 orderiui  in 
specialty  groups  in  a h tera/cliical  1 y ri?  lur.  i oi  gani/.-ation  which 
can  be  managed  by  maiern  rictiiods  of  Cerr-.'.aj.'  and  a ntrol. 

2.  The  new  personnel  structure  must  be  a<!a,  tabJ.-  ir,  aii  conceivable 
fornts  of  war  and  to  new  weapon  systems  . 

3.  It  must  proviUe  u ;iex;t.  ii-  i ’Mniework  . i-.ireer.s  and 

of  the  pay  sr-stem. 

4.  It  must  tie  .such  th  u 1 1 n,'  on;e.  !i‘  .ut;u-  .u.u  scale  of 

measures  necessary  to  : .Jv.u.ct:  the  solcii  , ■ . . ..  •■.■i  can  be 

rational.ly  derived  from  it. 

5.  It  is  to  provide  an  essentia]  preconuii  o . ii.-  oni  iral  integration 
of  the  armed  forces  in  society. 

This  is  ttie  overall  oti  jecLive  tor  a ne\7  perso-c;  ■ structure,  of  the  armed 
forces  from  win  ch  the  research  work  had  to  start  .'it.  iie.re,  our  first 
concern  was  to  ilcvelop  a set  of  survey  instrumcr.es  ch.it  would  satisfy 
two  requL remento  : First,  it  had  to  be  cap.ilil;  d ■ . i.iect  ing,  in  surveys 

to  be  conducted  ju  the  armed  forces,  ail  tl.e  data  neeueu  for  laaking 
structural  Improvements  within  the  framework  of  tb.e  objectives  mentioned. 
Second,  it  liad  to  be  such  th;it  it  could  In'  adndai  t ■.  red  by  the  Bundeswehr 
Itself.  This,  we  considered,  was  a sine  qua  non,  as  otlierwise  the  costs 
would  be  pri'hil)  1 L Lve  . 
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The  Federal  Ministry  of  Defense  commissioned  a scientific  institute  to 
develop  a system  for  job  description,  job  analysis  and  job  evaluation 
in  the  Bundeswehr . 

As  you  know,  there  have  been  developed  in  tlermany  and  in  other  countries 
quite  a number  of  methods  and  technqiues  for  job  analysis  and  job  descrip- 
tion. Having  examined  them,  however,  we  felt  that  none  of  the  procedures 
could  be  adopted  without  modification.  Most  of  the  procedures  had  been 
tailor-made  to  fit  the  specific  needs  of  the  industrial  organization  con- 
cerned and  are  not  suited  for  the  armed  forces  for  several  reasons. 

lb  mention  first  some  organizational  and  quantitative  aspects:  For  one, 

these  enterprises  are  generally  much  smaller  in  terms  of  the  number  of 
personnel  they  employ  than  the  armed  forces  of  the  Bundeswehr  with  their 
500,000  servicemen.  For  another,  in  no  other  enterprise  is  the  diversity 
of  jobs  anywhere  as  great  as  in  the  armed  forces.  And  finally,  the  enter- 
prise-tailored procedures  use  conventional  methods  of  job  analysis  and  job 
description.  Such  an  approach,  however,  is  not  feasible  for  the  armed 
forces  if  only  because  of  the  large  personnel  effort  that  it  would  involve. 

Apart  from  these  largely  organizational  and  quantitative  aspects  there  is 
of  course  the  peculiarity  of  the  duties  and  tasks  in  the  armed  forces  to 
be  considered  which  although  they  have  many  parallels  with  comparable  jobs 
in  the  civil  service  and  in  industry,  are  nevertiie I ess  distinct  from  them 
and  characterized  primarily  by  the  following  quality;  S.)ldiers  must  acquire 
skills  and  knowledge  and  possess  and  develop  abilities  which  are  to  be 
applied  only  in  a defense  emergency. 

The  evolvement  of  an  instrument  for  job  de.scriptiuu  and  jol-  analysis  was 
glared  to  one  central  objective;  to  cr.-..'  t!  , I"'  h >.  for  a system 

of  specialty  groups  under  which  all  position.',  .tu  nuhsumud.  Viiis  means, 
above  all,  that  answers  would  have  to  be  louu.i  to  tiie  .ul  lowing  four 
questions : 


1.  ilow  is  the  position  to  be  rank-orderi'd . 

2.  What  training  does  the  position  require . 

3.  Wliat  qualifications  and  abilities  doe.i  tin  .i,  .'  'on  .quite? 

4.  Vfliat  other  employment  potential,  vertical  and  lateral,  does  the 
incumbent  of  a position  offer? 

In  the  job  description  it  is  first  stated  wlnit.  dnt  ie.-!  tiie  incumbent  has 
to  perform.  Sucli  a breakdown  of  positions  into  duties  will  later  on  permit 
us  to  move  from  a distribution  of  duties  "as  is"  to  a distribution  of  duties 
"as  should  be".  The  duties  themselves  would  hardlv  be  changed,  only  their 
specific  content  and  combination  in  a job.  This  concept  of  duties  "as 
should  be"  is  also  reflected  In  the  designation  chosen  in  the  Bundeswher 
for  the  Instrument,  namely  "function  analysis".  Tlie  term  denotes  that 


what  Is  to  be  accomplished  is  not  an  analysis  of  tasks  as  actually 
performed  but  the  collection  and/or  development,  in  the  form  of  duties, 
of  system-related  functions  as  a basis  for  reorganizing  the  personnel 
structure , 

This  function-related  description  is  hence  oriented  not  so  much  on  what 
an  incumbent  actually  does,  than  on  the  result  to  be  achieved  in  the  form 
of  duties,  and  on  the  conditions  under  which  this  must  be  accomplished. 

To  summarize; 

Job  description  in  the  Bundeswehr  is  to  serve  four  primary  objectives: 

1.  To  enable  positions  to  be  groups  into  standardized  jobs,  and  these 
in  tuni  into  specialty  groups. 

2.  To  enable  requirements  profiles  to  be  developed  and  thus  a requirements- 
related  selection  of  personnel  for,  and  matching  of  personnel  with, 
specific  jobs  to  be  accomplished. 

3.  It  is  to  provide  sufficient  data  for  a requirements-oriented  training 
concept  to  be  determined. 

4.  It  is  to  furnish  the  basis  for  making  decisions  on  the  personnel 
structure  in  organizational  units. 

Here  the  problem  was  to  find  a job  element  suitable  to  meet  the  desired 
objectives  of  reorganization.  By  "job"  we  in  the  Bundeswehr  understand 
the  aggregate  of  the  duties  performed  in  a position. 

As  "duties"  we.  designate  the  smallest  organizational  elements  of  a job. 

Therefore,  the  instrument  was  so  designed  as  to  allow  the  positions  to 
be  broken  down  into  duties.  It  was  Important  that  the  different  contents 
and  typical  combinations  of  duties  were  inventoried  for  the  ultimate 
overall  survey.  In  terms  of  the  scope  of  the  undertaking  alone  this 
was  an  extraordinary  -let  me  say  quantitative  - problem.  The  scope  of 
the  problem  is  evident  from  the  following  data: 

The  personnel  classification  system  of  the  Buiiiieswelir  comprises  some  1,400 
specialties.  These  are  arranged  in  110  specialty  groups  and  these  in  turn 
in  23  specialty  are  "skill  level",  "shredout",  "occupation"  and,  finally, 
"position". 

The  "skill  level"  indicates  the  level  and  range  of  knowledge  and  abilities 
in  a specialty. 

The  "shredout"  identifies  part  of  a specialtv. 
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"Occupation"  designates  the  combination  of  a specialty  with  a skiji 
level  and  possibly  a shredout.  The  "position"  finally  identifies  the 
Job  in  the  organizational  structure. 

In  1971  and  1972  a number  of  preliminary  surveys  were  conducted.  They 
served  several  purposes.  Accordingly,  difierent  survey  techniques  were 
used.  The  objective  of  the  first  survey  phase  was  to  make  an  Inventoiy,- 
oi  the  various  job  elements,  i.e.,  of  the  dutie;:,  tasks,  knowledge  and 
skills  that  make  up  military  jobs. 

First  the  znetliod  of  open  questioning  was  used.  The  lespondonts  had  to 
complete  a three-part  questionnaire.  in  part  1 t:r  dr  1 i anil  el  tiieir 
duties  a.s  performers;  in  part  2 they  describe  i tlii  virious  tasks  required 
to  be  performed  in  execution  of  the  respective  ilit.i.,;  and  in  pai  t i tliev 
were  aske'i  to  list  job  factors  indicative  of  the  trii;.:  t.-quired  and 

the  type  and  degree  of  strain  Involved  in  the!  in*-.- . 

The  sur\'ey  was  conducted  by  specially  trained  i , ir.t  > rvi  ewers . 

Even  during  the  first  stage  of  the  survey  It  proved  neceb.ary  to  inter- 
view as  many  as  close  to  1,800  incumbents  from  aO  difftri'nt  units  of 
the  three  Ser\'ices, 

Next  followed  a methods  test.  In  it  tlie  first  draft  oi  a procedure  for 
surveying  duties  and  tasks,  especially  witli  regard  ti  allocating  "basic 
tasks"  to  diities,  was  tested.  At  this  point  let  me  give  you  a brief 
definition  of  the  terms  "duty",  "basic  task",  and  "ta.-'V"  as  we  vise 
them  in  the  Bnndeswehr,  "Duty"  we  call  that  coiiiponent  oi  a job  which 
can  be  derived  from  the  mission  and  which  pr  .-J.Je.s  tin-  li  v i.s  for 
determining  the  dlstrlbvition  of  the  responsibilities  associated  with 
duty  positions  and  hence  the  basis  for  rani.irg  them  ii,  t lie  organiza)  ional 
structure.  From  tlie  analysis  of  duties  typological  .iiarav  teri  st  ics  may 
be  derived  for  the  construction  of  requi  nmerts  and  qual  1 fi  cat  i iin  profiles. 
By  "basic  task"  we  understand  tliat  component  of  a )i)b  which  is  indispensable 
for  the  execution  of  a duty  in  a position  and  whiilv  provides  tlie  basis 
for  the  organization  of  training.  The  analy.sis  ol  basli-  tasks  yields 
occnpatlvmal/ technical  characteristics  for  (lie  c.uis  I I’uct  i on  (jf  reqnirements 
and  qualification  profiles.  As  "tasks"  we  li.sigi  ci  !ol)  i.lement.s  wliicJi 
describe  the  full  range  of  work  activities  in  a job,  lagaidless  of  wliether 
these  elements  are  particularly  vital  for  , !,■  e vi  > jit  -.cr  : the  diitv  (U‘ 

dut i es  or  not , 

Before  I continue  let  me  say  that  at  this  poir.t  ot  the  development  work 
we  found  ourselves  in  a blind  alley,  so  to  spe  ii  , 'or  li.  spite  intensive 
efforts  we  did  not  succeed  in  finding  an  anal'.tical  tool  wltli  which  we 
could  have  filtered  out  that  certain  ceiiiral  cfiraionent  nieefing  the  criteria 
of  a 'basic  task"  as  defined  above.  .kt  tills  pcint  tlie  development  work 
.succir, lilt'd  to  what  might  he  called  an  "atomi  zat  i 1 ( ' r* ' L*  s s " wliich  resulted 
from  the  sheer  overwhelming  flood  of  "ever  sm.ilicr"  job  elements.  Tiie 
r.eci>',s.ir''  "ivduction  work"  wiiiclt  therefore  ' oi  im  tlioilological 

reasons  had  to  ensure  that  now  at  least  al  I those  tasks  were  ascertained 
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which  were  similarly  indicative  of  the  requirements  for  occupational/ 
technical  training  and  the  occupational/technical  characteristics  to 
be  contained  in  requirements  and  quallf icatlon  profiles  as  the  vainly 
sought  "basic  tasks". 

At  the  same  time  the  methods  test  in  this  phase  of  the  work  also  per- 
tained to  various  scales  used  in  rating  the  time  spent  on  the  various 
duties  and  tasks  and  the  degrees  of  difficulty  and  strain  involved. 

On  the  basis  of  the  data  obtained  during  the  first  survey  phase  catalogues 
of  duties  and  tasks  were  compiled  for  four  selected  and  circumscribed  areas 
and  submitted  to  the  Incumbents  sampled  in  the  form  of  questionnaires.  ihe 
Incumbents  first  had  to  checkmark  all  duties  which  they  performed  in  their 
positions.  In  addition,  they  had  to  rate  the  average  time  spent  in  per- 
forming a duty.  Here,  too,  methodological  difficulties  were  encountered 
since  a mere  percentage  rating  cannot  do  justice  to  the  many  specific 
duties  which  may  occur  only  in  exercises  or  are  the  object  of  training 
in  instruction  courses. 

In  further  steps  the  respondents  had  to  check  all  the  tasks  they  performed 
in  their  position  and  subsume  them  under  the  various  duties,  as  well  as 
race  on  a scale  the  frequency  with  which  a task  occurred  in  a duty.  For 
methodological  reasons,  this  time  the  questionnaires  were  administered 
both  to  incumbents  as  performers  and  to  their  immediate  supervisors.  The 
supervisors  had  to  complete  the  questionnaire  not  for  the  duties  and  tasks 
of  their  own  positions  but  for  the  duties  and  tasks  of  their  subordinates, 
i.e.jof  the  same  individuals  who  had  previously  been  questioned  about 
their  jobs  as  performers. 

This  iin  thods  test  was  administered  to  240  performers  and  supervisors  of 
selected  positions  from  four  areas.  In  the  next  survey  phase  the  inventory 
of  duties,  tasks  and  other  job  factors  from  the  first  stage  was  comlined 
with  the  test-improved  part  of  the  survey  instrument.  This  led  to  t iie 
development  of  a catalogue  of  preliminary  functions,  liere  the  functions 
served  as  coarse  identifiers  of  duties.  They  represented  "siipra-duties" 
so  to  speak,  under  which  the  duties  formulated  by  the  inciimhents  had  to 
be  subsumed. 

These  function  questionnaires  contained  both  partly  standardized  as  well 
as  open  questions.  In  addition  to  subsuming  duties  under  a function  the 
incumbent  had  to  state  whether  he  performed  the  duty  himself  or  supervised 
its  performance.  The  function  questionnaires  pertained  only  to  the  self- 
performance of  duties.  If  an  Incumbent  supervised  the  performance  of  a 
duty  in  whole  or  in  part  he  had  to  describe  the  work  of  supervision  on  a 
separate  questionnaire.  The  surveys  in  this  second  stage  covered  some 
2,900  incumbents  from  65  different  units  in  the  three  Services. 
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Krom  the  experience  and  analyses  in  pliases  I and  2 resnlieri  an  inst  I'ument 
which  by  furnishing  a differentiating  description  of  duties  ami  other  fob 
elements  was  to  solve  tlie  problem  of  an  Armed  Forces- wide  uniform  Joi) 
description  sucli  that  all  relevant  criteria  were  covered.  The  purpose, 
in  a(i(iitlon  to  completing  tlie  functio-,  questionnaire,  >-.ms  lo  test  tin- 
practicability  of  the  instrument  in  terims  of  tlie  lime,  menral  and  physical 
effort  tiiat  respondents  could  reasonably  be  expected  to  expend  on  answering 
the  questions.  In  accordance  with  the  state  of  progress  of  the  development 
work,  all  questionnaires  this  time  contained  mostly  performul ated  answers 
so  that  the  work  of  the  respondents  was  largely  reduced  to  checkmarking. 

In  this  methods  test  some  180  Incutnhents  of  selected  positions  were  quest  lone 

In  a final  survey  phase,  the  purpose  of  which  was  to  fill  In  gaps  left  during 
survey  phases  1 and  2 and  thus  to  complete  the  systematic  survey  oi  jol' 
elements,  900  Inc'.mhents  in  some  20  schools  of  the  Bundeswehr  were  questioned 

This  stage  proved  necessary  for  two  reasons;  In  reviewing  each  ol  the 
preceding  surveys  it  had  been  found  that  the  selected  samples  liaii  iu.Vei 
been  fully  cov'ered,  be  it  that  "stand-ins”  with  other  (iian  the  desired  job 
characteristics  were  encountered,  or  be  it  that  the  desired  numl.'er  of 
incumbents  of  certain  positions  could  not  be  reached.  These  "gaps"  had 
to  be  filled  in.  But  beyond  that  - and  tliis  wa.s  aiiol'er  consideration  - 
out  purpose  in  extending  this  surr’cy  to  sciiools  w.as  to  obtain,  through 
liiis  ''oc.it  1 onaJ  ly  and  technically  especially  qualifed  group  of  iridi'’i<iuals , 
an  indirect  validation  of  the  completeness  of  previously  ascertained 
duties  and  other  job  elements. 

Ilf  re.sult  of  the  preliminary  surveys  lo  d.'  oi.ip  ...  • i • t.ir  jol 

.les(  I ipt  ion  can  l e suramnrized  as  follows:  Frcra  i lu-  ..  i.ng  of  4,i.'oo 

incnmliiuit s insulted  a collection  cf  65,0i1i)  data  on  uu'  ios  md  280, odn  h.-u  ,i 
on  tasks. 

•n.  a.tiige,  eacn  i iic.umbent  listed  betwoon  ! ..  , ; : ■.  anl  in 

nnu  80  tasks. 


'iat.!  on  duties  and  tasks  had  to  bt-  ..|  iug  . . .mp.-i  i .-il , . le.te. 

• iiul  .wiMuged  later  on.  For  that  pur,  os.'  t ht  , we  ri- , oi  i. '.  , ..sit  ion 
stparately  .ind  together  with  the  posltion'.s  other  eliar  e t erlsti  cs , coded 
.iinl/or  textuallv  documented  and  stored  on  'iii:'.":  ' . , hu  eiiginal  (iat  .a 

from  the  preliminary  surveys  are  of  cons  Me  r.ab  1 ('  i nnori  .nve  inasmuch  as 
the''  rerlect  the  language  oi  the  performers  of  the  d.nt  iiis  t liem.se  Ive.s . 
Theretore,  it  i.s  advisable  that  this  language  .should  not  he  ccerlv  edited 
in  the  reviews.  The  more  distinctly  the  langu.age  ot  the  performers  is 
pieserved  in  the  subsequent  statements  of  duties  and  tasks,  the  greater 
will  be  the  reliability  of  the  results. 

Analysis  of  this  great  quantity  of  survey  data  is  aC'  Oinplished  by  means 
ot  programs  developed  for  that  purpose,  which  permit  the  data  to  be 
assemliled  and  sorted  as  desired.  A detailed  categorical  coding  system 
which  can  be  used  not  only  for  the  data  from  the  surveys  but  also  for 


othur  source  material  now  enables  duty  catalogues  to  be  compiled  tor 
certain  specialty  groups  as  an  Instrument  for  Job  description  and  Job 
analysis , 

After  the  preliminary  surveys,  the  job  description  instrument  was 
designed;  it  consists  of  two  parts. 

The  first  part  "Instrument  Part  A"  is  composed  of  catalogues  of  question- 
naires for  the  various  specialty  groups  to  question  incumbents  as  per- 
formers. Each  of  these  catalogues  of  questionnaires  is  organized  into 
four  sections  with  the  following  content: 

Section  1 is  a questionnaire  to  ascertain  personal  background  data 
(of  the  incumbent)  and  job  satisfaction.  This  section  is  the  same  in 
each  catalogue  of  questionnaires. 

Section  2 constitutes  the  main  part.  It  contains  questionnaires  on  the 
duties  whicli  are  performed  by  the  Incumbent  himself  and/or  whose  performance 
he  supervises.  The  scope  of  section  2 is  of  necessity  variable.  It  is 
determined  by  the  number  of  duties  in  the  respective  specialty  group.  The 
duties  in  section  2 are  classified  into  various  duty  groups,  namely: 

- Duties  which  are  typical  of  the  respective  area,  l.e.  specialty-group- 
rt‘)  a ted ; 

- geiieial  administrative,  duties,  i.e.,  duties,  which  are  not  restructed  to 
a certain  area; 

- s iipe  i-'i  sory  duties,  i.e., duties  of  incumbents  with  super'risory  functions 
will  ill  occur  more  or  less  in  all  areas; 

- 1.  . ,1 ' ),  1 A.  r.iui'ous  duties;  these  az'e  considered  to  be  duties  whlcli 

pi'operly  nelong  in  other  areas  but  which  are  also  performed  by  incumbents 
in  I lie  respective  area  (for  Instance  voice  radio  communicat ions  in  the 
field,  lienee  a duty  properly  belonging  in  the  area  "communications", 
wuieii,  however,  must  be  performed  by  the  members  of  an  armored  infantry 
sqnaii  since  in  tills  subarea  no  position  "voice  radio  communication" 
exists. 

Sect  ion  i contains  questionnaires  on  vocational/teclinical  training  and 
further  training  whicli  is  conducted  within  the  scope  of  the  job. 

Section  4 finally  contains  questionnaires  on  direct  technical  supervision 
of  personnel  in  the  performance  of  their  duties. 

In  addition,  eacti  catalogue  of  questionnaires  comes  with  detailed  instruc- 
tions regarding  tlie  answering  technique.  The  answers  to  the  questions  are 
not  entered  in  the  questionnaires  themselves  but  are  given  on  separate 
response  sheets  and  continuation  sheets.  To  complete  the  response  sheets 
respondents  checkmark  preformulated  answers.  The  continuation  sheets 
are  provided  for  Incumbents  to  write  in  information. 
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Since  the  administration  of  this  instrument  by  the  Bundeswehr  itself 
is  not  so  easy  and  since  also  questions  of  data  analysis  had  to  be 
verified,  the  testing  of  the  •'inetrument  of  job  description  was  conducted 
in  the  following  four  specialty  areas  this  year: 

- infantry  duties 

- aircrew  duties 

- navigation 

- personnel  administration. 

I will  now  show  you  on  some  vlewgraphs  sections  of  "Instrument  Part  A": 
(Vlewgraph  1) 

These  questions  pertain  to  the  personal  background  of  the  Incumbent  and 
his  satisfaction  with  his  job.  Specifically,  the  items  covered  are; 

1.  the  highest  level  of  formal  education  attained 

2.  the  service  status  of  the  individual 

3.  the  type  of  career 

4.  length  of  service  in  present  duty  position. 

And : (Vlewgraph  2) 

5.  Present  rank. 

Then  follow  a few  questions  concerning  the  incumbent's  attitude  toward 
his  position,  such  as: 

6.  the  interest  he  takes  in  his  present  job; 

'/.  his  contentment  with  his  pay,  considering  his  training  and  quallfica- 

t tons ; 

8.  wiieiher  he  can  apply  hls  abilities  In  his  job; 

d,  ni;,  coni  fiumcnt  with  hls  pay,  considering  the  strain  Involved  and  the 

perl ormance  required; 

10.  . oiiipai  I s.in  with  other  jobs; 

11,  la  with  similar  jobs  in  respect  of  pay. 
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(Vlewgraph  3) 


This  vlewgraph  shows  a duty  from  the  area  of  "infantry  duties".  The 
questions  pertain  to; 

- the  various  tasks  to  be  performed  in  the  duty; 

- the  circumstances  under  which  these  tasks  must  be  performed  (difficulty); 

- the  frequency  of  occurrence,  and 

- t he  importance  of  the  duty. 

The  answers  are  entered  in  a separate  response  sheet  which  I will  shov.’ 
you  in  the  next  vlewgraph. 

(V'lewgraph  4) 

This  is  the  sheet  in  which  the  answers  to  the  questions  contained  in  the 
catalogue  of  questionnaires  are  entered.  The  number  of  the  respective 
question  is  marked  on  the  response  sheet  by  making  an  oblique  stroke 
across  the  corresponding  number  on  the  response  sheet.  Only  in  '3  cases 
must  the  respondents  write  figures,  all  other  notations  are  made  by  placing 
an  oblique  stroke  in  the  appropriate  box.  This  facilitates  the  work  of 
the  respondents  and  reduces  the  number  of  illegible  entries. 

(Vlewgraph  5) 

These  questions  pertain  to  the  vocational/technical  training/f urtlier 
training  wliich  the  incumbent  conducts  in  the  units.  The  purpose  of  the 
questions  is  to  establish  the  degree  of  strain  Involved  in  the  position. 
Specifically,  the  questions  pertain  to: 

- the  number  of  individuals  trained; 

- tlie  importance  of  the  training  (whether  the  objective  is  earning  a 
certificate  of  qualification,  a permit,  a license,  an  authorization, 
ce rt i f I cat  ion  of  a certain  level  of  prof iciencv) ; 

- on-the-iob  training; 

- training  througli  academic  instruction  (frequency); 

- ascertaining  the  training  result; 

- amount  of  extra  preparation  required  on  the  part  of  the  incumbent; 

- importance  of  this  <iuty  relative  to  the  incumbent's  other  duties. 

So  much  for  the  first  part  of  the  job  description  instrument,  or 
"Instrument  Part  A". 
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( V i <_■  Av;r;i;.r. 


Transfoj 

re  tiie 

Charaoteristioo  of  the  individual 
(asalgnee)  and  vocational  satiaf action 

> 

Task-No 

Che 

ck  all  t) 

ve  follow  in 

questions.  ILark  ths 
applicable  answer  I.'o  on 
the  data  acquisitioi’ 

sheet.  , 


1 • Elrhfist  educational  level 
oc'^  ir  .~g 

Special  elementary  school  ...  1 

Prlnnry  ochool  without  quali- 
fication examination  or  equi- 
valent educational  level  ....  2 

Primary  school  v?lth  qualifl- 
! cation  examination  or  equi- 
valent educational  level  ....  5 

Secondary  ochoo]  ordinary 
lovol  or  equivalent  educa- 
tional level  4 

Tnchnioftl  Colic f.f  v*th  quall- 
f lost  ion  exuir.ir.a  lion  5 I 

I Sacondary  school  i^radi.atlon  - | 

: advancad  level  or  equivalent 

i educational  level  6 

; Uirher  proi’cnaional  school 

I with  qualification  exaalna- 

1 tion  7 

j Technical  Colic "o  with  quali- 

j fication  ezn.tin.ition  8 

I Ifnivernlty  education  i 

' (diploma)  9 j 


10 


12 

13 

14 


I 2 , ’ n 

j Cor.i'orlpt  

Lo;if5-'tor.‘  -vie omen 
i (21  loonthc)  

I 7A5.ng-tcrci  psrricciaen 

I ( 2 i'caru)  

i I/Or>(-’-terr!  oorvicemen 

{ ( 3 years)  

j ''ro’  or  rvice:;en 

j ( A yc.-rs  ) . 


Long-term  aei^ice^isB 
( 5 to  7 years)  15 

Long-term  eervicooen 
( 8 years)  16 

Long-term  servicemen 
( 9 to  11  years)  17 

Long-term  servicemen 

(12  years)  13 

L-ng-terr.  eerviconon 

(13  to  14  years ) 19 

Long-term  servicemen 
( 15  years  ) 20 

Hegular  soldier  (20  years)  ..  21 

Regular  soldier  22 

3 > Career 

Administrative  career  .......  23 

Uedical  Eorvico  24 

iillltary  Land  25 

Kllitary  Geography  26 

unitary  specialist  27 

^ loh q h Fi-ve  you  b ^ f n in 

your  re  .-'T;!  poaiTi'oni 

Lees  than  3 months  28 

3 to  6 months  29 

7 to  12  months  50 

1 to  2 years  j 1 

more  thori  2 years  52 


, V] 


) 


5. 


in  vovr  T^rosfint  rank 
or  cr''irar  4 1. ivQ  r.^n.-:  'i* 

Prlvat#  (OR  1 ) 

PrlYat#  (02  2)  

Privato  let  Claes  (OR  3) 

Prlvat*  1st  Claes  (OR  4) 

Corpoi'al/Pc-Vty  Officer 
’i'd  Olaea  (OR  5)  

2cr£oa:it/r-aity  Officer 

Claaa  (OH  6)  


Ser'^cont  1st  Class/Petty 
Officer  let  Claaa  (OR  7) 

liaater  Ser^rfeant/Chief  Petty 
Officer  (OR  0 ) 

Scnior’Chicf  Petty  Officer 
(OR  6)  

liaater  Sor,^eant/Chief  Petty 
Officer  (OR  9)  

2nd  Lieutenant/Enaign  .... 

1st  Lioutenar.t/Licutonant 

Ccptain/Lieutu.Tart  ....... 

Ua Jor/Lleutcnant  Conrander 

Lieutenant  Colonel/Connander  51 
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34 

35 

36 

37 

40 

41 

42 

45 

44 

46 

47 

48 
5o 


Colonel/Captaln 


Prigadier  General/Hear 
Adnlral  (lowpr  half)  ., 


J-’ajor  Genaral/Kear  Adslral 
( uppex’  Ixalf  ) . . . 

Lieutenant  General/ 

Vice  Adijiral  

Oaneral/Adulral  


G.  yn';’'  t^i-ininn  regarding 


I find  r,iy  prcnnnt  activity 
eery  in teror.liiig  

prcdoisinantly  interesting 

partially  interoating  .... 

not  intereoting  


54 

55 

56 

57 
50 


7 . Conaldoring  ay  training 
quail . I cat  ion  i ua 


and 


oatiof ied  

loba  satisfied  . . 

not  aatiafied  

with  ny  pay 

8,  In  ny  preaent  activity  rv 
aoililioo  are 


hardly  utilized  

partially  utilized  .... 
predominantly  utilized 
complotsly  utilized  ... 


j / < QV|n  concerning 


59 

60 

61 
62 


9.  Conaldorlng  the  denanda  and 
pe  ri  or: : c e eaTTea  for  in  ny 

p r e u e n /"a  c tTv  Tt  y 

I am 

satiofied  

less  GBtiofied  

not  eatiafiod  

with  ciy  pay. 

1 o . A d i f f r_r •~nt  ao t Ivi t y than  ny 
prs.'u-nt  ■-■cuivicy  vrouTa 

surely  give  me  more  satio  - 
faction  

pOBBibly  give  me  more  eatla- 
faction  

presumably  not  give  mo  more 
satiaf action  

definitely  not  give  me  more 
eatisi'action  

f1  1 . I_iy  oi  •"!  r a r 1 e 0 n_v.'  slrllnr 

I Fli'  v1~' ' h r 

i ry  T-i.'o;  ;-nr  activ'l  fy'!.!-.  p:\r~i~ 

■ I Cl  lo'.v 


yes 
no  . 
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64 

65 


66 

67 

68 
69 


70 

71 

72 


73 

74 

75 

76 


77 

76 


(v  Lir.v  T.'iph  >) 


Controlling  the  camouflaging 
of  sites  and  objects 


Only  if 


Tranefere  the 


pppl Icable 


Camouflaging  of  sites  and  objekcta 


lonlv 


japplicable 


j 7fi0lc-i\o 

Tranefere  ti.e 

4 Tatk-No 
t 

i^b' 

UM 

1 


uneck  all  the  following 
queationa.  Uark  the  appli- 
cable answer  No  on  the  data 
acquibltion  sheet. 


/' 


1 . 


2. 


ffhat  do  you  cnipyiflage  ? 

weapons  and  equipment  . . 

bunkcrc  

land  vehicles  

air  vehicles  

water  vehicles  

firs  positions  

buildings,  installations 
facilities  


hi  o h ca'Tiouf  lagi n g moans 
bo  use  ? 

terrain  exploitation  and  selec- 
tion of  location  (shadow, 
conctaling  covei-,  ground  confi- 
guration)   0 

natural  oamouf laging  means 
( grarr.i , foliage  , busnes , bi-anchco  ) 9 

protective  coating  against 
infiMi-ed  and  radar  1o 

artificial  ceann  of  camouflage 
(camonflfiga  netn,  tenting, 
artificial  fog)  ii 

provloionary  camovif Inglng  moans  12 
* -V ’ll:  '-nuflage  do 

camouflage  against 


noiee  13 

rieiblllty  14 

light  19 

infrared  16 

radar  17 


At  what  1 evel  do  v o u 
accompTisl:  ;.-.is  tasit  ? 

a)at  the  level  of  normal  dutiee 


several  times  dally  138  i 

once  daily  159  ! 

three  tinea  weekly  14o  I 


one  to  two  times  weekly  ...  141  | 

one  to  three  times  monthly.  142  i 

.eso  than  once  monthly  ....  143 

never  137 

b)withln  courses  or  during  i 

exercises  j 

more  than  12  weeks  annanually  132 


up  to  12  weeks  annanually  133 
1 to  4 weeks  annnually  ...  134 
lean  than  1 week  annanuHl ly  139 
never  136 


V ’i  Tjt  H 1 rr,  ^ ■'1  c a nee  ha  a t hi  r 
'•'1;  {TT  c'j  .ri  r:on  u tT'  otnar 
1 1 r'r.n  0^  y'  . ■'  r o-'iti>..n 

celect  the  applicable  value  on 
tho  scale  between  1 and  7 


1 vary  Impotant  main  task  . . 

2 

3 important  task  

4 

9 lean  important  task  

6 

7 comparatively  insignificant 
secondary  task  


144 

149 

146 

147 

148 

149 


uBgwi  rnn 


I Control  of  opaclttllred  and  advanced  . 
j trainln<^  dr.  tn»  unite  ^ 


Only  If 


applicable 


Speclaliaod  ani  advanced  trainlnij 
In  the  unite 


iOnly 


If 


applicable 


Tranofere  t^.e  corre- 
opendlng  taok  No  fror„ 
the  ATB  list 
(aTB  means 

Training  Activity 
5e  signet  ion ) 

Transfere  the  appiic^o'.  • 
task  Ko  from  tne  AT5 
Check  all  the  follcv.:.., 
questions  the  ap,l.- 

cable  answer  No  or,  the 
data  acquisition  sheet 


! 1,  Ko«  many  persons  from  this  ao- 
; tivlty  do  you  provide  apociallzod 

i and  advanced  training  simultane- 


ously? 

up  to  three  1 

four  to  eight  2 

nine  to  fifteen  3 

ulxtean  to  thirty  A 

mors  than  thirty  5 


2,  lo  the  training  to  be  concluded 
by  reaching  a certain  training 
level,  attaining  a permission, 
licenss  or  authorization? 
no,  no  special  qualification  ..  6 


attaining  a permission/  11- 

I osnse/  authorization  7 

qualification  certificate  6 

reaching  a training  status j 

status  rt  ^ORs)  9 

status  7 lNC''a)  iO 

status  6 i Sergeants ) 11 

status  b (.Master  Sergeants)  ..  12 

status  2 (Officers)  ...  13 

3.  Do  you  provide  on-the-^ob- 
trainlng,  l.e.  during  normal 
I duty  hours? 

j no  ...........................  14 

I yeo  15 

\ 

i 

!4,  Have  you  to  perform  practical 


sxsrcisea  apart  from  your  nor- 
mal functlone  in  order  to  pro- 
vide personnel  from  this  acti- 
vity Bf'Ociallzsd  and  advanced  • 


training? 

no  16 

y««  17 

Do  you  give  theoretical  in- 
struction during  the  opeclal- 
ited  and  advanced  training? 

no  18 

yea,  daily  ig 


yea,  two  to  throe  tlmse  weekly  20 
y«a,  at  least  ones  weekly  ....  21 
yes,  less  than  ones  weekly  ...  22 


6,  In  what  form  do  you  control 
the  training  status  of  this 
personnel? 

no  control  necessary  23 

oral  tests  (questions)  24 

essays  25 

written  final  examination  ....  2o  ' 

observation  during  performance 
of  work  27  ' 


7.  Does  the  specialized  and  ad- 
vanced training  of  this  per- 
sonnel call  for  epeclal  prepa- 


rations by  you? 

no,  not  necessary  26 

provision  of  instructive  and 

illustrative  material  29 

preparation  of  syllabus  and 
sequence  of  syllabus  30 


8.  What  significance  has  this  task  i 
in  comparison  with  other  tasks 
of  your  position?  ‘ 

Select  the  applicable  value  on 
the  scale  from  1 to  7 
1 very  important  main  tosk  ..  144  i 


2 U5 

3 important  task  146  i 

4 147 

5 less  important  task  148 

6 149 

7 comparatively  unimportant 

secondary  task  150 


( 

i 

I 


f 


Mi.t  now  a lew  words  concerning  the  second  part,  "instrument  Part  B". 

In  this  part,  not  the  performers  of  the  duties  themselves  are  questioned. 
Rarlier,  after  ttie  data  from  survey  part  A liave  been  analysed  the  dui  ies 
in  their  various,  ascertained  modalities  are  submitted  to  experts  to 
have  tliem  verify  which  knowiedge  and  skills  on  the  one  hand  and  whlcli 
abilities  on  the  other  hand  are  required  for  the  performance  of  the 
respective  duty  in  that  certain  modality.  We  are  hence  concerned  with 
ascertaining  the  requirements  that  are  associated  with  the  performance 
of  duties,  that  is  requirements  in  terms  of  knowledge  and  skills  tliat 
can  he  acquired  on  the  one  hand,  and  the  relatively  constant  personality 
traits,  hence  the  abllltle''  which  must  be  regarded  as  a prerequisite 
for  the  successful  acquisition  of  a certain  knowledge  and  of  certain 
ski  1 Is . 

'1  he  testing  of  tl\is  instrument  in  the  Bundeswolir  began  in  dune  of  tliis 
year.  In  tills  test  the  catalogues  of  questionnal  res  for  tlie  four  specially 
groups  mentioned  above,  namely  infantry  duties,  aircrew  duties,  navigation, 
and  personnel  administration,  t re  used.  Witiiln  the  scope  of  survey  part 
A In  the  four  areas  mentioned,  some  3,200  servicemen  at  60  duty  stations, 
who  had  been  selected  in  accordance  with  a certain  selection  plan,  were 
questioned  by  12  teams. 

Extensive  preparations  were  required.  The  briefing  of  project  attendants 
of  the  Services  as  well  as  of  the  survey  teams  took  a total  of  4 days. 

In  addition,  it  proved  expedient  to  provide  the  survey  teams  with  a 
12-page  guide  to  assist  them  in  properly  motivating  the  servicemen  to 
be  questioned. 

in  this  pilot  survey  the  servicemen  questioned  completed  a total  of  24,000 
response  slieets.  On  average,  tite  Incuml^ents  questioned  responded  on  d; 
duties.  The  variance,  however,  was  quite  considerable.  It  ranged  fror, 
a maximum  of  98  to  a minimum  of  9 duties. 

At  this  time  the  testing  of  "Instrument  Part  B"  has  not  yet  been  tully 
completed.  The  findings  from  this  test  run  will  help  determine  the  course 
to  be  followed  by  the  Bundeswehr  in  its  endeavor  to  reform  its  personnel 
St  met  tire. 
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Possibilities  and  Limitations  of  vlob  Analysis  and  Job 
Evaluation  in  Armed  Forces 

Lieutenant  Colonel  Seuberlich 
German  Army 

Let  me  begin  with  a preliminary  remark.  At  one  of  the  many  meetings 
which  of  late  have  been  held  to  discuss  analytically  and  critically 
questions  of  performance,  the  Minister  of  Economics  of  the  Federal 
Republic  of  Geniiany  said,  in  May  1973,  among  other  things  that: 

The  performance  motive  of  some  was  sometimes  considered  as  a pretext 
for  the  profit  and  power  motive  of  others.  It  also  served  as  an 
expression  of  ureed  and  the  striving  for  power,  as  few  gave  orders  and 
many  obeyed.  Also,  individual  work  performance  in  a highly  industrialized 
economy  was  not,  or  only  indirectly,  measurable.  Performance  was  not 
measured  by  the  market  value  of  labour  but  by  the  market  value  of  the 
product.  This  inability  of  performance  to  be  measured  was  apparent, 
among  other  thinns,  from  the  fact  that  wages  and  salaries  were  mostly 
fixed  by  collective  bargaining,  that  is  independent  of  individual  per- 
formance. Replacing  the  performance  principle  by  some  other  principle, 
however,  would  not  remove  the  negative  aspects  for  which,  incidentally, 
the  performance  principle  was  not  the  cause.  Independent  of  the 
collectively  negotiated  wane  structures,  which  had  to  be  retained  in 
the  economy,  there  was  the  possibility  for  the  individual  to  be  rated  on 
his  performance.  However,  the  Minister  conceded,  an  objective  evaluation 
mechanism  was  not  yet  known.  In  this  connection  it  was  also  necessary 
that  nsychol oqi cal  1 y-moti vated  performance  readiness  be  accompanied  by 
performance  ability,  which  in  turn  presupposed  equality  of  opportunity 
for  all,  which  again  could  be  provided  only  by  an  efficient  educational 
system. 

From  all  this  it  is  already  apparent  to  what  extent  tne  "mystery  that  is 
man"  - mysterious  with  respect  to  the  interactions  between  requi rements , 
performance  and  evaluation  - constitutes  the  determining  factor,  and 
this  factor  will  be  the  principal  point  of  reference  throughout  my 
presentation.  When  I say  "the  mystery  that  is  man"  I use  that  expression 
to  allude  to  the  difficulty  of  predicting  human  behaviour  and  point  to 
one  of  the  reasons  why,  for  instance,  entrepreneurs  concentrate  their 
planning  and  goals  predominantly  on  the  management  of  inanimate  things. 

As  soon  as  man,  however,  becomes  the  object  of  planning,  psychological, 
work-physiological,  and  sociological  aspects  become  dominant.  They 
confront  planners  with  a host  of  bewildering  determinants  for  every 
planning  model.  The  problem  here  lies  in  the  ascertainment,  quantifica- 
tion and  also  qualitative  determination  of  all  the  essential  factors 
which  influence  the  behaviour  patterns  of  man  and  thus  make  possible 
prognostication  and  planning. 


With  reqarcl  to  the  questions  that  pertain  to  armed  forces  we  have  to 
keep  one  thing  foremost  in  mind;  the  behavior  of  man  cannot  be  predicted 
with  absolute  certainty,  it  is  never  static  but  a correlating  product  - 
subject  to  constant  change  - of  the  personality  structure  of  Liie  ii.d  . , : • 
and  the  relevant  environmental  situation  in  which  he  finds  himself.  Of 
course,  nobody  today  would  deny  that  there  are  test  instruments  wiui  a 
sufficient  degree  of  reliability,  and  meaningful  tables  of  norms  which 
have  proven  themselves  to  be  useful  instruments  of  prognostication . By 
using  such  instruments  the  Hamburg  police  school,  for  instance,  was  able 
to  reduce  the  number  of  their  police  school  dropouts  from  an  initial  10 
percent  to  3 percent.  And  in  the  endeavors  to  develop  the  basis  fot  a 
new  personnel  structure  in  the  German  Armed  Forces  the  problem  or  per- 
formance prediction  will  soon  also  gain  central  importance.  If  the  means 
of  psychodiagnosis  are  to  be  employed  merely  for  the  purpose  of  detectinu 
extra  educational  capacity  in  efforts  to  control  the  develo^rment  of 
talented  individuals  and  to  systematize  the  advancement  of  co-wori'ers, 
this  does  not,  in  principle,  pose  any  serious  difficulties  since,  it  is 
true,  a time  span  of  45  minutes  is  sufficient  to  determine  wif.h  ulequctte 
accuracy  the  intelliqence  quotient  of  35  individuals.  Rut  here  the 
question  already  arises  as  to  what  is  actually  expressed  in  the  intelliue 
quotient?  What  kind  of  intelliqence  is  measured? 

By  a definition  datinu  from  1923  intelligence  is  simply  what  is  meusuied 
in  the  intelliqence  test.  This  definition,  as  is  widely  agreed,  has 
lonq  been  overtaken  by  developments.  More  acceptable  to  us  today  is  a 
definition  from  1963  according  to  which  "intelliqence  is  the  abilu  of 
an  individual  to  think  in  concrete  or  abstract  verbal,  numerical  ai'id 
space-ti'.t  terms;  it  enables  him  to  deal  successfully  with  many  comp’.  - 
and  with  ttie  aid  of  particular  groups  of  abilities  - also  quite  specific, 
situations  and  tasks." 

This  definition  already  takes  into  acc(/unt  the  findinns  of  modern 
intelligence  research  which  holds  that  intelliqence  cannot  lie  expre^sen 
in  a rigid  number.  And  what  is  more,  intelligence  here  is  evoressly 
related  to  the  successful  dealing  with  complex  and  also  very  specific 
situations  and  tasks.  And  this  brings  us  to  the  heart  of  our  subject. 

As  early  as  1969  the  Manpov/er  Structure  Commission  apnoinfed  by  th*' 

Federal  Minister  of  Defense,  determined  as  a result  of  its  first  ii  vest  i - 
gabions  : 

The  concept  ot  the  standard  officer  and  standard  non-commissioned  officer 
with  a broad  utilization  spectrum  and  broad  interchangeability  can  no 
longer  be  kept  up  on  a modern  personnel  structure.  This  obsolete  concept 
must  be  replaced  with  a modern,  vocation-  and  f unction-rel  ated  cotU'-pt. 

In  particular  it  must,  through  service-peculiar  differentiations,  t'e  in 
consonance  with  the  multiple  division  of  labor  in  modern  armed  torcis. 
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The  new  vocation-  and  function-related  concept  of  the  modern  soldier 
will  have  repercussions  in  many  areas.  It  will,  amonq  other  things, 
have  to  stress  a duty-related  typology  of  the  soldier.  Quite  revealing 
in  this  respect  is  the  attitude  of  the  Air  Force  who  hold  that  in  the 
Air  Force  everybody  today  is  a specialist.  In  this  connection  it  is 
interestina  to  note  the  following  aporoximate  distribution:  only  5 
percent  of  all  jobs  are  directly  combat-oriented  while  95  percent  may 
be  subsumed  under  the  headings  of  technics,  economics,  or  administration. 
These  latter  jobs  can  also  largely  be  compared  with  corresponding  civilian 
jobs.  But  regardless  thereof,  standard  criteria  must  be  developed  for 
the  representation  and  categorization  of  the  great  variety  of  occupations 
in  the  armed  forces.  Special  attention  must  in  this  context  be  given  to 
the  adequate  recognition  of  civilian  vocational  background  knowledge  and 
experience,  especially  for  appointment,  advancement  and  career  progression 
purposes. 

This  finding  is  based  above  all  on  comprehensive  polls  of  young  men  of 
draft  age.  It  appeared  that  their  attitude  toward  the  Bundeswehr  was 
influenced  in  a negative  way  as  soon  as  they  included  their  own  situation 
in  their  consi derati ons . This  applied  to  those  who  took  a generally 
positive  view  of  the  Bundeswehr  as  well  as  to  the  qrouo  whose  overall 
attitude  to  the  Bundeswehr  was  negative  to  begin  with.  It  is  worth 
noting  that  this  trend  also  exists  among  potential  NCOs  although  this 
group  shows,  on  the  whole,  a much  more  positive  affinity  with  the 
Bundeswehr.  The  greatest  shift  in  attitude  was  noted  among  high  school 
graduates.  Attitude  research  confirmed  that  it  is  above  all  practical 
considerations  which  determine  their  actions,  for  instance  their  decision 
to  enlist  beyond  their  statutory  term  of  service.  From  this  it  is  evident 
what  consequences  a negative  shift  in  attitude  as  a result  of  his  practical 
consi derations  can  have  for  an  individual's  actual  behaviour.  These 
prarticol  considerations  include  in  particular  the  career  nrospects 
that  DOtential  draftees  think  the  Bundeswehr  can  offer.  Only  one  fifth 
of  those  polled  rated  their  career  chances  in  the  Bundeswehr  as  clearly 
positive.  The  majority  of  the  respondents  who  by  their  education  were 
eligible  for  an  officer  career  do  not  believe,  for  instance,  that  to 
become  an  officer  requires  special  qual i fi cations . Although  of  the  two 
target  grouns  (potential  NFOs  and  potential  officers)  nearly  one  third 
of  those  polled  (32  percent  and  27  percent  respectively)  do  in  principle 
consider  volunteer  service  in  the  Bundeswehr  to  be  a possible  career 
alternative,  this  is  not  saying  anything  about  the  actual  recriiitment 
potential;  what  is  decisive,  rather,  are  t h«  concrete  reasons  for  wtiich 
potential  IlCOs  and  -I'^ficers  vjould  be  willinn  to  volunteer.  Here  it  is 
found  that  rational  considerations  determine  their  readiness  to  volunteer. 
Financial  security  (45  ).  qood  vocational  training  (35  ),  good  chances 
of  career  progression  (IQ  ) ranoe  foremost  among  the  positive  motivations 
whereas  "liking  the  military  profession"  motivates  only  7 percent. 
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One  of  the  central  problems  of  personnel  management  is  hence  utilization 
consistency.  This  is  the  degree  of  congruity  of  knowledge  and  abilities 
on  the  one  hand  and  the  job  with  its  concrete  duties  on  the  other.  A high 
degree  of  congruity  not  only  increases  work  efficiency  but  also  enhances 
the  individual's  subjective  contentment  in  his  job.  An  analysis  of 
utilization  consistency  in  the  Bundeswehr  showed  a notable  discrepancy 
with  utilization  consistency  in  the  civilian  sector.  Only  12  percent  of 
the  conscripts  and  only  26  percent  of  the  temporary  volunteers  feel  that 
in  their  present  jobs  in  the  Bundeswehr  they  are  employed  according  to 
their  qualifications  and  abilities.  Only  in  the  group  of  regular  soldiers 
is  the  degree  of  utilization  consistency  much  higher:  no  less  than  56 
percent  consider  being  employed  in  keeping  with  their  knowledge  and  abilities. 
But  even  here  the  discrepancy  with  the  corresponding  figure  in  the  civilian 
sector  is  still  14  percent,  the  consistency  value  in  the  civilian  sector 
having  been  established  at  70  percent. 

In  evaluating  this  situation  we  must  of  course  keep  in  mind  the  particular i f , 
of  the  mission  of  the  Bundeswehr  and  as  a result  thereof  consider  that 
military  service  has  certain  characteristics  which  altiiough  they  nuiv  also 
be  encountered  in  other  areas  are  - because  of  their  combinaMon  and  weight  - 
nevertheless  significant  for  the  armed  forces.  Soldiers  must  not  only 
accept  certain  abridgements  of  their  individual  freedoms;  they  must  also 
daily  prepare  to  use  force  and  learn  to  operate  into  situations  of  danger. 
These  particularities  which  are  function-determined  and  irrevocabltr  are 
also  to  be  clearly  shown  by  job  analysis  as  being  indispensable  for  the 
mission,  in  order  that  they  may  be  duly  taken  into  account  in  the  assignun 
of  pay  and  service  grade. 

This  is  a difficult  task  to  accomplish,  given  the  present  pay  and  grad* 
rules  in  effect  in  the  Bundeswehr,  rules,  which  are  oriented  chiefly  oi 
traditional  criteria  and  which  have  only  partially  been  adiusted  to  r.hang“d 
internal  and  external  conditions. 

Among  the  major  structural  elements  of  the  Bundeswehr  which  ..al.e  scmviiui 
its  personnel  problems  difficult  is  not  just  the  al  ready-n'entioned,  out- 
dated concept  of  the  standard  officer  and  standard  noncommi  ssione<l  .t  ici:r, 
the  very  principle  of  which  is  at  variance  with  the  growing  di fferentiat ion 
of  job  requirements.  Directly  associated  therewith  is  tlic  sys'ei 
summary  rather  than  a requi rements-related,  i.e.,  analytically  determinable, 
job  evaluation.  And  finally  there  is  the  structure  of  the  position  pyramid 
which  also  does  not  allow  for  the  necessary  degree  of  differentiation  but 
which  is  rather  also  in  the  nature  of  a "standard  position  pyraudd'. 

Thus  the  main  elements  of  the  present  personnel  structure  *>e  o'"  ra  i .ized 
by  criteria  which  operate  to  make  the  filling  of  existing  positions  with 
adequately  qualified  personnel  and  the  assurance  of  an  adequate  performance 
readiness  on  the  part  of  recruited  personnel  a latent  problem. 
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let  us  now  n ' '>►'*  'lance  at  the  comnley  nf  evaluation  and  pay. 

Here  the  lack  ot  precise  military  job  profiles  entails  a variety  of 
disadvant  d(jes  since  the  present  evaluation  and  pay  system  is  based  on 
rank.  Accordinuly  it  is,  by  its  nature,  linear  in  structure  and 
relatively  little  adaptive.  Thus,  the  major  tools  of  a modern  personnel 
structure  policy,  i.e.,  the  assinning  of  pay  and  service  qrades , are 
severely  restricted  in  their  potential.  But  nothing  thati  the  structure 
of  a specific  jol>  profile,  i.e.,  the  presence  of  specific  job  charac- 
teristics in  a certain  constellation,  can  be  the  criterion  for  the 
conjunction  of  positions  and  grades. 

In  some  specialty  areas  the  discrepancy  between  job  and  service  grade  was 
ascertained,  from  the  investigation  it  was  obvious  that  a redistribution 
of  many  enlisted  nv-n  presently  in  NCO  grades  to  other  grades  was  necessary. 
Thus,  for  d 1 percent  of  the  NCOs  in  the  specialty  area  of  administrative 
duties  the  s'/ndrome  of  duties  comes  closer  to  that  of  lower  ,'anks  than 
to  the  iob  profile  of  higher  service  grades.  This  was  only  an  example. 

In  connection  with  these  analyses  another  finding  deserves  to  be  mentioned. 
The  subjective  importance  of  improper  grading  with  respect  to  its  effect 
on  job  satisfaction  is  different  from  what  it  had  been  suspected  to  be. 
Neither  are  soldiers  in  higher-rated  positions  especially  content  with 
their  pay  nor  are  soldiers  in  lower-rated  positions  especially  dissatisfied. 
This  result  confirms  that  satisfaction  with  pay  is  determined  rather  by 
factors  of  time  pressure  and  work  load  and,  overall,  the  degree  of  strain 
involved  in  u jot).  In  the  problematical  group  of  NCOs,  for  instance,  much 
more  than  in  any  other  group,  utilization  in  higher-qualified  but  not 
corres()ondingl y hi nher-pay i ng  positions  is  translated  into  discontentment 
with  their  chances  of  career  progression. 

In  evaluating  this  problem  area  we  must  also  consider  the  factor  of  job 
satisfa<  lion  in  rnniunction  with  the  utilization  cnnsistencv  of  personnel. 

It  has  beet)  -hown  that  *‘he  factor  "fighter  activity"  exerts  the  strongest 
inflijonrr-  on  utilization  consistency,  because  with  the  growing  content 
of  fighter  duties  in  a job  less  and  less  civilian  vocational  skills  and 
abilities  can  be  applied.  Here  the  peculiarity  of  the  mission  of  the 
Cundesvvehr  is  clearly  apirarent.  There  simply  is  no  civilian  equivalent 
for  fighter  duties  in  the  stricter  military  sense  of  the  word.  This  means 
that  by  and  large  only  a small  percentage  of  personnel  in  this  area  find 
that  degree  of  job  “satisfaction  which  is  required  for  a positive  concept 
of  self.  It  is  above  all  the  tyne  of  individual  who  quite  unreflectedly 
thinks  that  "being  a soldier  is  fun".  The  negative  influence  exerted  by 
fighter  duties  on  utilization  consistency  can,  however,  be  largely  offset 
by  the  strain  imposed  by  extreme  work  conditions  as  well  as  by  a greater 
measure  of  communuative  activity,  especially  if  these  two  factors  can  be 
combined.  In  addition,  activities  of  an  organizing  and  planning  nature 
are  apt  to  reinforce  utilization  consistency.  All  in  all,  this  is  an 
area  wliich  requires  further  investigation. 
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Also  to  be  fm^ntioned  in  tin's  context  is  the  attempt  by  the  Bundeswehr  to 
reqard  discipline  and  personal  freedom  as  not  being  mutually  exclusive. 

It  was  undertaken  to  redefine,  under  the  term  "Innere  Fuhrunq"  (inner 
conduct  and  leadership),  the  concept  of  command  and  leadership  under  the 
changed  conditions  i.ntaining  when  the  new  German  armed  forces  were  organized 
beginning  in  1955.  Here  especially  three  factors  conie  into  play  wiiich  are 
sufficiently  relevant  also  in  the  context  of  evaluation  to  be  briefly 
mentioned  here; 

1.  A certain  apprehension  that  German  militarism  might  possibly  be  revived 
was  dispelled  by  the  lawmaker  and  the  Bundeswehr  leadership  when  it  was 
decided  that  man  as  the  center  of  our  political  and  social  system  should, 
also  as  a citizen  in  uniform,  remain  at  the  center  of  our  defense  efforts. 

2.  Along  with  the  rest  of  technology,  arms  technology  had  advanced  so 
dynamically  that  also  for  this  reason  new  modern  fomis  of  training, 
leadersiiip  and  tactics  were  needed.  Thus,  the  situation  was  and  is 
that  no  superior  can  acquire  the  whole  ranoe  of  diversified  special 
knowledge.  Rather,  he  has  to  rely  on  the  conscientiousness  and  compe- 
tence of  qualified  subordinates. 

3.  The  threat  to  the  free  world  was  not  restricted  to  the  threat  of  force. 
Therefore,  also  the  imral  struggle  had  and  has  to  be  considered,  in 
which  the  soldier  needs  knowledge,  judgment  and  inner  strength  to 
sustain  him. 

From  this  point  of  view,  every  military  suierior  is  also  an  educator  in  a 
special  area.  This  is  a constant,  central  task  since  even  in  peacetime 
spiritual  and  moral  forces  often  determine  the  value  of  a soldier  more 
decisively  f-ftan  do  sf/ocialized  skills.  Linders tandinq  fo)  the  duties  and 
the  rnnsri on  I i '.n  th  which  they  are  performed  detep'drio  the  state 

of  trainincj  of  the  troops  and  the  operational  readiness  of  the  weapon 
systems.  Thu'^  , the  deterrent  effect  of  the  Bundeswehr  depends  at  all 
times  on  the  sum  of  tlie  activities  which  the  soldier  performs  in  (;eace- 
time.  In  the  Operations  Manual  of  the  German  Army  it  says;  "War  is  the 
severest  test  of  man's  moral  and  physical  strength.  In  war,  moral  qualities 
are  therefore  oft.en  more  important  than  intellectual  abilities  . . . 

The  vdstness  and  ei.iptiness  of  the  battlefield,  the  tremendous  effect  of 
firepower,  ttie  rapidity  of  all  maneuvers  and  the  absence  of  fixed  fronts 
more  than  over  (.all  f(;r  soldiers  who  show  individual  initiative  and 
responsibility  and  who  act  in  furtherance  of  the  overall  objective." 

This  quotation  clearly  describes  the  aims  for  the  soldier's  education 
in  the  Bundeswehr.  The  importance  of  these  aims  in  all  armed  forces 
throuohoi/t  tfie  world  is  uncontested.  But  their  attainnrnt  is  possible 
only  to  a limited  extent  and  hardly  less  difficult  than  a systematic 
analysis  of  the  respective  requirements  to  be  met  by  the  individual  and 
their  corresponding  evaliiation.  The  psychological  factors  which  lie  in 
conflict  here  are  too  heterogeneous.  Consider  for  instance  the  situation 
when  panic  broke  out  1914  in  the  battle  near  Gawai ten-Gumbinnen  as 
described  by  Hesse  in  his  "General  Psychologos". 
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The  Army  Onerdt.inns  Manual  calls  this  phenomenon  a sudden  breakdown  of 
moral  resistances.  It  can  be  brouqht  on  by  overwhelm! nq  combat 
experiences  as  well  as  by  seemingly  insignificant  or  unrelated  even!-  . 
but  also  by  psychological  measures  of  the  enemy.  Stouffer,  the  well- 
known  American  military  sociolonist  comes  to  the  conclusions  that  personal 
motives  and  relationshi ps  do  not  alone  determine  the  combat  ef *'oct'''  ' • ■ 
of  fiqhtinq  units.  Officers  and  enlisted  men  must  be  motivatoi!  to  L1..1I 
end.  The  best  criterion  on  which  to  base  a prediction  of  behavior  in 
combat,  according  to  Stouffer,  is  the  individual's  attitude  toward  his 
insti  tutional  i zed  role  which  says  that  he  is  a soldier,  not  a civi'''an. 

The  social  role  confronts  him  with  the  alternative  of  either  fight  r 
bear  the  institutionally  envisioned  consequences. 

But  we  know  that  in  spite  of  intensive  efforts  no  system  has  as  yet  been 
devised  that  allows  us  to  predict  which  soldiers  would  actually  fight 
and  which  would  just  trudge  along.  Behind  all  this  is  the  question  of 
defense  motivation  which  is  of  vital  importance;  for  the  readiness  to 
commit  oneself  up  to  the  point  of  self-sacrifice  is  not  a matter  of 
course.  The  "what  for"  must  be  strong  enough  to  overcome  the  existential 
anxiety  in  man.  The  "what  for"  must  also  provide  the  impulse  for  the 
individual's  physical  commitment  and  loyal  performance  of  duty  in  the 
daily  routine  of  peacetime  military  service.  For  the  individual  soldier 
the  meaning  or  senselessness  of  his  actions  can  be  derived  only  from  the 
"what  for".  Only  a strong  "what  for"  motive  makes  for  conviction  and 
leads  to  an  insight  into  and  acceptance  of  the  special  conditions  of 
military  life  that  are  so  different  in  many  ways  from  those  of  a civilian 
existence. 

The  requirement  for  initiative  and  responsibility  on  the  part  of  the 
individual,  down  to  the  last  man  who  is  a specialist  in  a certain  function, 
follows  from  what  has  just  been  said. 

With  the  aho'"’  aspects  in  mind  the  author  developed  a proce.luie  liy  which 
he  tested  1,200  conscripts  immediately  after  they  were  inducted  for  militrtr 
service  in  order  to 

- determine  how  the  aptitudes,  attitudes  and  abilities  of  raw  conscripts 
correspond  to  the  special  requirements  of  service  in  the  Bundeswehr; 

- ascertain  the  characteristics  by  which  groups  of  various  educational 
attainii\ent  differ  from  each  other; 

- establish  criteria  for  the  formation  of  realistic  training  classes; 

- ascertain  relevant  characteristics  for  grouping  individuals  in  different 
training  classes,  considering  individual  aptitude  patterns. 

At  first,  the  overall  objective  was  to  find  a military  intelligence 
quotient.  This  goal,  however,  had  soon  to  be  abandoned  as  the  number  of 
variables  was  too  great  and  as  too  many  different  evaluation  factors  had 
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to  be  applied  so  that  in  operationalization  it  proved  imoossible  to 
combine  and  express  the  necessary  differentiated  statements  in  one 
numeric  quantity.  On  the  other  hand,  it  turned  out  to  be  quite  feasible 
and  also  much  more  expedient  to  express  the  findings  and  values  in  a 
military  intelligence  profile.  The  test  procedure  was  adapted  to  the 
specific  situation  of  the  young  people  prior  to  their  inriiiction  tor 
military  service. 

The  test  was  conducted  during  the  so-called  "enrollment  processing  phase" 
of  the  recruits,  that  is  to  say  during  a period  when  the  influence  exerted 
by  instructors  was  still  minimal.  Hence,  the  test  reflects  almost 
exclusively  the  raw  recruit's  original  equipment  in  terms  of  aptitudes, 
skills,  knowledge  and  views. 

The  test  was  divided  into  two  main  parts.  The  academic  part  comprised 
40  main  questions,  each  containing  up  to  20  subquestions.  The  questions 
had  to  be  answered  under  time  pressure  and  were  arranaed  in  such  erratic 
sequence  that  a pattern  could  not  be  recognized  by  the  testees.  The 
practical  part  contained  twelve  main  tasks,  each  with  up  to  six  sub-tasks, 
some  of  which  had  to  be  solved  at  night  in  unfamiliar  terrain.  In  the 
evaluation  the  results  of  the  academic  part  and  of  the  practical  part 
were  interrelated  in  a multiplicity  of  ways  and  then  consolidated  into 
18  overall  grades  for  each  individual  soldier.  In  all,  the  five  following 
main  areas  with  their  subdivisions  were  evaluated  (viewgraph  1): 


Viewgraph  1 

1 . Physical  abilities 

- sports  performance 

- taxability  and  endurance 

- reaction 

2.  flental  abilities 

- technical  understanding 

- orientation,  and  grasp 
of  terrain 

- concentration  and  receptiv- 
ity 

3.  Personality  traits 

- purposiveness 

- readiness  to  commit  self 
and  to  assume  responsibilities 

- decision 

- leadership 
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4 . Rel ati onshi ps  with  the  environ- 
ment 

- attitude  towards  fellow  men 
and  companionship 

- prior  relationship  with 
superiors  (parents,  teachers, 
pri nci pals , etc. ) 

- concept  of  military  superiors 

5 . Civic  knowledge  and  attitudes 

- civic  knowledge 

- perception  of  the  Communist 
threat 

- attitude  to  the  notion  of 
fatherl and 

- attitude  to  the  notion  of  a 
free  democracy 

- attitude  to  military  service. 
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The  consnlidated  test  results  show  a military  intelligence  profile, 
developed  and  measured  on  conscript  recruits  inducted  in  1966  and  broken 
down  by  educational  groups,  and  presented  in  the  18  areas  mentioned.  1 
will  now  show  you  two  segments  (viewgraph  2): 

The  grade  scale  contains  values  1 - 3,  each  with  nine  intermediate  values. 
The  grades  denote; 


1 .0  = fully  sui table 
1.5=  sui table 

2.0  = conditionally  suitable 
2.5  = not  yet  sui table 

3.0  = unsuitable 

The  horizontal  line  at  2.0  indicates  the  borderline  between  suitability 
and  conditional  suitability;  it  may  be  called  the  military  minimum  require- 
ment which  the  young  soldier  must  have  exceeded  by  the  end  of  basic  training 
in  order  to  meet  the  subsequent  requirements. 

The  educational  groups  are  identified  by  letters  as  follows; 

VB  = elementary  school  leavers  (9  years  of  schooling  with  subsequent 
vocational  training)  (303) 

A = high-school  graduates  (13  years  of  schooling)  (100) 

M = high-school  and  commercial  school  leavers  (10  years  of  schooling) 

(100) 

V = elementarv  and  special  school  leavers  without  vocational  training  (31) 

0 = average  of  persons  surveyed  (534) 

Mental  abilities  (viewgraph  2) 


With  a score  of  2.0  the  technical  understanding  of  the  high-school  graduates 
is  already  suitable,  that  of  the  unskilled  workers  (elementary  school 
leavers  without  vocational  training)  on  the  other  hand,  at  2.4,  still 
very  much  in  need  of  development.  In  technical  understanding  the  conscripts 
are,  on  average,  brought  up  relatively  quickly  to  the  requirements  of 
training. 

Also  in  the  area  of  "grasp  of  terrain"  (orientation  in  and  perception  of 
terrain,  e.g.  , estimating  distances,  utilizing  terrain  features,  selecting 
a position)  the  group  of  high-school  graduates,  at  2.0,  distinguish  them- 
selves by  a suitable  performance  and  also  during  the  subsequent  training 
showed  themselves  generally  more  capable  of  development  than  the  soldiers 
of  the  other  groups.  But  even  for  these  the  distance  to  the  dotted  line 
can  be  closed  within  a few  weeks  from  2.2  to  a satisfactory  2.0. 
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The  poor  test  perfomance  in  concentration  and  receptivity  is  symptomatic 
of  the  generally  noticeable  fluctuating  intelligence  even  of  high-school 
graduates,  caused  by  the  flood  of  stimuli  of  the  modern  urban  world  which 
with  television,  cars  and  neon  signs  has  invaded  even  the  remotest 
villages.  While  it  is  possible  to  eventually  close  this  gap  it  reguires 
a lot  of  patience,  good  didactics  and  toughness  on  the  part  of  the 
instructor.  With  the  average  performance  being  2.5,  this  is  not  so  much 
a question  of  intelligence  than  of  education  and  the  environment.  Even 
the  high-school  graduates,  having  scored  2.1,  are  initially  on  average 
only  conditionally  up  to  difficult  concentration  tasks.  A rating  of  1 
was  achieved  by  only  4 percent. 

. Now  as  for  personality  traits  (viewgraph  3): 


To  determine  purposiveness  in  a short  test  is  possible  only  to  a very 
limited  extent  since  only  disposition  and  tendencies  but  not  the  decisive 
factor,  namely  perseverance  of  will  over  an  extended  period  of  time,  could 
be  tested.  Seen  in  this  light  the  test  result,  as  shown  in  the  viewgraph, 
is  only  of  limited  value.  On  the  other  hand,  the  high  average  score  of 
1.5  reflects  our  "open,  permeable"  performance-oriented  society.  In  other 
words:  in  our  society  it  is  worthwhile  for  everyone  to  set  goals  for  him- 
self. Whether  he  attains  these  goals  depends  primarily  on  his  own 
capabilities,  his  efficiency,  knowledge,  perseverance,  etc.  - in  short, 
on  his  own  performance.  The  degree  to  which  he  uses  his  potential  is 
a question  of  intelligence,  energy  and  endurance.  It  deserves  to  be 
noted  that  this  fact  is  apparently  also  well  known  among  unskilled 
workers  (elementary  school-leavers  without  vocational  training)  although 
they,  probably  as  a result  of  their  past  experiences,  show  less  "courage" 
to  set  themselves  ooals  than  the  other  groups. 

In  summary,  the  test  result  permits  this  conclusion  to  be  drawn  which  is 
of  particular  importance  for  the  military  leader:  the  young  soldier  is 
receptive,  he  is  responsive;  it  depends  essentially  or,  the  skill  of  the 
military  superior  to  keen  this  responsiveness  alive,  to  foster  it  and  to 
transform  it  into  willing  independent  cooperation. 

The  readiness  of  an  individual  to  commit  himself  and  to  assume  responsibility, 
too,  can  be  determined  in  a test  only  to  a limited  extent  although  in  this 
area  substantially  more  secured  information  could  be  gained  through  practical 
tasks.  But  the  degree  of  reliability  of  the  information  must  still  be  rated 
as  highly  "conditional"  as  in  the  final  analysis  no  one  (an  say  with 
absolute  certainty  of  himself  how  he  would  react  sometime  in  the  future  in 
some  situation  or  other  under  variegated  specific  conditions.  His  behaviour 
depends  ultimately  on  a great  number  of  factors  and  circumstances,  not  the 
least  among  them  being  the  individual's  "condition  of  the  day"  if  we  want 
to  call  it  that,  which  may  be  determined  to  a larger  or  lesser  degree  by 
hunger,  fatigue,  physical  injuries,  etc.  Nevertheless,  the  average  score 
of  slightly  over  2.0  indicates  a suitable  basis.  Among  the  young  conscripts 
the  readiness  to  assume  responsibilities  and  to  commit  self  is  widely 
distributed.  It  is  fully  sufficient  as  a basis  on  which  to  build  and 
secure  training  and  education. 
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With  an  average  rating  of  nearly  2.0  the  decision-making  power  of  ttie 
conscripts  may,  from  the  very  first  day,  be  called  sufficient.  The 
graph  shows  three  interesting  peculiarities: 

- The  group  of  the  high-school  graduates  does  not  rank  first.  It  obtained 
the  same  average  mark  of  2.0  as  the  high-school  and  commercial  school 
leavers  with  10  years  of  schooling  and  the  unskilled  workers.  The  reason 
may  be  seen  in  the  fact  that  because  of  their  greater  inental  agility 
high-school  graduates  are  aware  of  many  more  possibilities  than  their 
lesser  educated  peers.  For  the  high-school  graduate  the  result  is  a 
certain  hesitancy  - he  weighs  and  considers,  often  much  longer  than 

the  others,  before  he  finally  makes  his  decision. 

- The  group  of  the  elementary  school  leavers  with  subsequent  vocational 
training  is  the  only  one  which,  with  a mark  of  2.1,  stays  below  the 
dotted  line.  They  are  the  only  group  among  the  testees  who  have  both 
a completed  vocational  training  and,  on  average,  comparatively  longer 
experience  in  jobs  of  somewhat  increased  responsibility;  in  other  words 
they  are  a little  more  aware  than  the  others  of  the  practical  burden  of 
responsibility,  and  this  is  perhaps  why,  on  average,  they  hesitated 
somewhat  more  than  the  others  in  making  their  decision. 

- The  unskilled  workers  seem  to  have  apnroached  the  decision  tasks  pretty 
"nonchalantly",  and  in  doing  so  scores  astonishingly  well. 

In  summary  it  may  be  said:  mind,  education,  and  decision  are  interrelated 
in  a mariner  which  is  hard  to  objectivate.  In  the  lower  range  of  military 
requirements  for  decision-making  the  level  of  education  and  knowledge 
plays  only  a minor  part.  The  natural  aptitude  of  the  average  conscript 
in  this  area  fully  meets  the  requirements. 

What  has  been  said  of  deci sion  applies  analogously  to  leade rship . In  all 
educational  aroups  there  are  conscripts  in  sufficient  number  who  possess 
enough  original  aptitude  for  minor  leadership  tasks.  A certain  superiority 
of  the  group  of  high-school  Graduates  and  high-school  and  comntercial  school 
leavers  with  10  years  of  schooling,  however,  is  already  apparent.  Practical 
experience  during  training  bears  out  this  finding.  The  superiority  of  these 
two  groups  increases  as  leadership  demands  get  higher.  This  does  not  mean, 
however,  that  individual  other  soldiers,  especially  among  the  group  of 
elementary  school  leavers  with  subsequent  vocational  training,  could  not 
also  achieve  the  same  results.  The  top  performers  in  the  VB  group  often 
even  surpass  the  average  high-school  graduate. 

Let  me  add  that  leadership  qualities  were  tested,  among  other  things,  in  a 
simulated  crisis  situation,  for  groups  of  five  testees  each  in  order  to 
determine  - to  the  extent  that  this  is  possible  at  all  - social  spontaneity 
in  group-dvnami c interaction. 


618 


But  let  us  now  consider  the  possibilities  of  evaluation  on  the  basis  of 
three  general  criteria; 

A.  Jobs  of  the  executing  type,  that  is  jobs  involving  predominantly 
executing  responsibility. 

B.  Jobs  of  a predominantly  - "tactical"  - nature,  as  I will  cal)  it, 
characterized  as  a rule  by  a mixture  of  executing  and  directing 
responsibi 1 i ty. 

C.  Jobs  of  a predominantly  "strategic"  nature,  characterized  by  an 
outstanding  degree  of  directing  responsibility. 

In  jobs  of  group  A incumbents  are  told  what  has  to  be  done  and  how  to 
do  it.  An  incumbent  of  this  type  of  job  must  sticl<  as  closely  as  possible 
to  regulations  in  order  to  satisfy  the  regui rements . Although  he  may 
suggest  changes  he  is  not  authorized  to  direct  any.  Incidentally,  job 
descriptions  for  positions  of  this  type  can  be  drawn  up  by  what  mioht  be 
called  job  description  technicians  in  accordance  with  established  guidance 
or  even  by  the  incumbents  themselves.  Most  job  descriptions,  as  we  know, 
are  of  this  type. 

Not  so  in  the  case  of  group  B.  In  this  category  belong  all  those  positions 
or  jobs  of  the  lov;er  or  middle  management  and  of  specialists  whose  main 
task  is  to  plan,  direct  and  supervise  the  activities  of  the  executing 
incumbents.  Generally,  therefore,  only  the  results  to  be  achieved  are 
prescribed  for  the  B-group  Hence,  incumbents  are  expected  to  work  out 
for  themselves  whatever  measures  are  needed  to  achieve  these  results. 

The  content  of  such  a position  is  hence  determined  to  a considerable  extent 
by  the  incumbent  himself.  Within  certain  limits  he  can  freely  apply  his 
specific  knowledge  and  faculties.  Therefore,  methods  of  job  description 
and  job  evaluation  developed  with  a view  to  the  A-group  are  of  necessity 
no  lonner  fullv  applicable  to  jobs  in  the  B-group.  In  tho  jnh<^  of  this 
group  tfie  multiplicity,  heterogeneity,  complexity,  and  chanqing  priorities 
of  the  duties  interact  in  such  a way  that  the  method  of  listing  duties  and 
tasks  is  no  longer  sufficient. 

The  jobs  of  a '--(lomi  nantly  "strategic"  nature,  finally,  are  top  positions. 
Characteristical ly,  these  are  the  positions  where  it  is  determined  what  is 
to  happen  to  an  organization  in  the  long  run,  what  aims  it  is  to  pursue, 
where  it  is  headed.  A strategic  position  is  therefore  shaped  largely  by 
the  incumbent.  Here  th-  complete  unfolding  of  the  incumbent  and  his 
identification  with  the  job  must  be  expected  and  demanded.  Therefore, 
the  mere  description  of  a strategic  position  alone  regui res  an  extra- 
ordinary degree  of  empathy  with  management  functions  and  understanding 
for  problems  involving  a high  degree  of  directing  respons ibi 1 i ty . 
Description,  analysis  and  evaluation  of  such  jobs  is  for  the  majority 
of  presently  applied  methods  a test  which  only  very  few  are  likely  to  pass. 
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In  spitP  of  the  difficulties  indicated  even  large  organization  of  the 
size  and  complexity  of  modern  armed  forces,  which  in  the  area  of  personnel 
organization  have  embarked  on  job  description,  analysis,  and  evaluation, 
are  likely  to  try  to  include  the  entire  organization  in  a uniform  descrip- 
tion and  evaluation  system.  If  only  because  of  the  high  personnel 
fluctuation  on  the  one  hand  and  of  rotation  on  the  other  any  vertical  or 
horizontal  limitation  of  whatever  kind  in  the  system  must  be  ruled  for 
the  armed  for.es.  Otherwise  a marked  decline  in  urgently  required  per- 
formance motivation  would  have  to  be  expected  since  the  requirement  for 
structural  visibility  and  transparency  could  no  longer  be  adequately  net. 

For  all  the  reasons  mentioned  the  Geneva  system  of  work  evaluation,  at 
least  as  far  as  the  four  basic  requirements  are  concerned,  should  tie 
applied  in  principle  also  to  job  evaluation  in  armed  forces.  In  pursuance 
of  this  aim  the  Bundeswehr  has  commissioned  within  the  framework  of  its 
basic  personnel  structure  studies  a "job  and  employment  standard"  (TVS) 
to  be  developed  as  a specific  structural  instrument  of  organization.  The 
job  standards  are  designed  to  cover  the  typical  duties  of  the  positions 
in  their  typical  modalities  as  well  as  the  associated  requirements  profile. 
They  thus  constitute  the  descriptive  part  of  the  TVS.  They  can  be  grouoed 
and  dealt  with  under  such  aspects  as  personnel  structure,  training,  personnel 
managuneiu  and  personnel  regeneration.  This  is  the  first  step.  In  a next 
step  the  jub  standards  are  then  compared  with  each  other  for  similarity 
and  with  respect  to  the  level  of  required  qualifications  in  order  to  develop 
criteria  for  the  establishment  of  a system  of  specialty  groups.  Here  the 
statements  arc>  each  referred  to  a job  standard  and  presented  in  an  employ- 
ment stamiard.  Ti)e  combination  of  the  two  statement  complexes  results  in 
the  job  and  eig  .uyment  standard,  the  TVS, 

The  importani:-.  nf  jolt  and  employment  standards  extends  into  all  areas  of 
personnel  organ ization , planning,  management,  and  training,  because  in 
all  areas  prf.blem  solutions  must  be  based  on  data  and  assumptions  concerning 
the  content',  of  j^  bs. 

The  coti  <• . .1  . Lunsists  of  12  parts.  Parts  1 and  2,  with  <‘heir  job 

designati-ip  (ATI’.)  and  otlier  identifying  references  are  to  enable  general 
class ' Fit. .1  .III,  t.iidt  is  the  systematic  arrangement  of  individual  positions. 
Parts  3 and  4 serve  the  purpose  of  ordering  duties  according  to  executing 
and  directimi  responsibility  and  by  centrality.  Parts  5 and  6 list  the 
principal  abilities  and  skills  required  and  the  necessary  school  and 
vocational  edijcation.  Part  7 describes  the  required  career  background 
of  incumbents,  differentiating  between  civilian  and  military  prior 
employment.  Furthermore , reference  is  made  to  training  courses  which  are 
considered  either  to  be  beneficial  or  requi red  for  successful  performance 
of  the  job.  Parts  8 and  9 show,  especially  for  first  assignments,  the 
rquirements  in  terms  of  aptitude  and  suitability  and  for  subsequent 
assignments  in  terms  of  capability  and  experience.  Part  10  deals  with 
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the  .:n.l  horizontal  military  and  civilian  assignirent  potential 

(the  tI '■  ! sulisequent  career).  Parts  11  and  12,  finally,  contain 
data  vhiicn  are  derived  from  duties  and  requirements,  such  as  grade  of 
pay,  minimiii'i  time  in  a job,  etc. 

It  is  envisaged  that  the  TVS  contain  a maximum  of  twelve  requi rements 
character! sti cs  whose  occurrence  in  the  qualification  profile  of  an 
individual  contribute  to  the  successful  performance  of  the  duties  of  the 
standard.  These  characteristics  must  be  rank-ordered.  The  rank-order 
is  derived  from  the  various  criteria  of  the  duty  and  from  the  requirements 
analysis.  As  for  the  subsequent  career,  the  determination  of  the 
possibilities  of  horizontal  and  vertical  employment  of  personnel  is 
based  on  the  same  methodical  premises  as  the  determination  of  the  preceding 
career. 

I now  come  to  the  conclusion  of  my  presentation: 

The  objective  of  function  analysis  within  the  scope  of  the  groundwork 
for  a new  personnel  structure  of  the  Bundeswehr  calls  for  standards  whose 
statements  are  so  differentiated  that  all  associated  positions  can  be 
uniformly  dealt  with  under  the  aspects  of  personnel  organization,  personnel 
management,  personnel  planning  and  training.  It  follows  from  this  require- 
ment that  by  reason  of  a factually  correct  rank-ordering  of  the  individual 
position  also  the  evaluation  of  that  position  and  thus  its  pay  grade  is 
fixed.  The  grade  of  pay  will  ultimately  be  determined,  however,  not  only 
by  tiie  I haracf eristi cs  of  the  job  but  also  by  the  experience  and  per- 
formani.f'-  of  the  individual  incumbent.  Rank,  by  contrast,  cannot  be  geared 
to  tht_  (.erson  oi-  his  performance  but  must  exclusively  be  determined  by  the 
characteristics  of  the  job,  namely  by  the  requirement  for  visible  expression 
of  directing  responsibility. 

We  know  that,  a long  and  difficult  road  is  ahead  of  us  until  this  objective 
is  achieved. 
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Job  malysis  and  its  related  procedures  have  become  fairly  common 
practices  in  many  organizations  as  a means  of  providing  personnel  planners 
and  managers  with  information  about  the  nature  and  requirements  of  on-going 
work.  Because  of  sharply  rising  costs  and  the  seemingly  inevitable 
concomitant  manpower  reductions,  such  occupational  information  has  recently 
assumed  an  even  greater  importance  within  military  services  around  the  world. 
More  efficient  personnel  selection  and  recruitment,  classification  and 
placement,  training,  personnel  appraisal,  job  evaluation,  and  jiib  design  are 
absolute  necessities  as  the  size,  composition,  and  function  of  tne  various 
armed  forces  come  under  closer  and  closer  scrutiny  by  their  governments. 

Although  the  Royal  Australian  Air  Force  (RAAF)  has  long  recognized 
the  need  to  know  more  precisely  what  its  airmen  do  on  their  jobs,  it  has  only 
been  in  the  past  five  years  that  a systematic  endeavour  for  collecting, 
analysing,  and  reporting  job  information  has  been  undertaken.  Based 
primarily  on  techniques  reviewed  overseas  during  an  investigation  of  Royal 
Air  Force,  Canadian  Defense  Forces,  and  United  States  Air  Force  occupational 
research  programs,  the  RAAF,  in  1967,  initiated  a modest  job  analysis  and 
training  evaluation  effort.  During  the  next  two  years,  two  job  analyses 
were  accomplished.  Unfortunately,  these  studies  were  conducted  by  a job 
analyst  as  an  additional  duty  rather  than  as  a primary  job,  and  in  addition 
they  suffered  from  being  different  in  design  and  approacli,  with  the  survey 
instruments,  response  forms,  and  rating  scales  varying  on  eacii  occasum. 

This  lack  of  standardization  proved  especially  costly  in  both  job  analyst 

and  computer  programmer  time  as  unique  EDP  programs  were  required  to  analvse 

each  return.  However,  the  situation  was  understandable  as  these  surveys 
represented  Lite  first  attempts  to  introduce  job  analysis  into  the  RA\F  and 
was  the  natural  result  of  trying  to  find  the  'best  wav'. 

in  the  meantime,  the  RAAF  was  seeking  clocci'  co-oper.it  ion  v.'itii  the 
USiVF  througli  the  formulation  of  an  exchange  program  in  tlie  job  analysis 

area.  Tliis  exchange  became  a reality  in  June  1970  witli  the  assignment  of  a 

RAAF  squadron  leader  (major)  education  officer  to  the  Air  Force  Human 
Resources  Laboratory  in  San  Antonio  to  study  occupational  researcli  technicjues 
and  operational  job  analysis  procedures.  At  tlie  same  time,  tlie  USAF 
exchange  officer,  a captain  psychologist,  was  jiosted  to  Headquarters  Support 
Command  in  Melbourne  to  develop  and  supervise  tlie  implemontat ion  of 
appropriate  job  analysis  and  training  evaluation  methodologies  within  llu' 
RAAF. 


Thus,  the  RAAF  system  of  job  analysis  described  in  this  paper  is  a 
moditied  version  of  Lae  USAF  job  inventory  approach  pioneered  over  the  last 
decade  bv  the  United  States  Air  Force  Personnel  Research  Laboratory,  now  a 
division  of  toe  Air  Force  Human  Resources  Laboratory. 


■ , •;  . : JOJeaul'es 

Although  the  job  inventory  approach  to  occupational  analysis 
(Morsh,  l‘hiO;  ,‘lorsh,  !9b4;  Morsh,  Madden,  and  Christal,  1961)  is  well  known 
to  most  of  you,  we  would  like  to  briefly  summarize  the  RAAF  version.  First 
there  is  Liie  I'ons  truction  of  the  job  inventory.  As  you  know,  the  job 
inventory  is  composed  of  two  basic  sections:  background  information  questions, 

and  a list  of  the  significant  tasks  performed  by  airmen  at  all  levels  of  rank 
and  job  prijficiency  within  a mustering  (USAF  specialty).  An  initial  listing 
of  tasks  is  constructed  from  available  published  sources  such  as  manuals, 
regulations,  training  documents,  etc.  This  list  is  then  subjected  to  an 
interview  review  by  senior  NCO  subject  matter  specialists,  and  to  a mail 
review  by  additional  senior  NCO  specialists  working  in  the  field.  The 
preliminarv  listing  of  tasks  is  then  revised  in  light  of  their  comments  and 
recommendations  for  changes,  additions,  or  deletions,  and  the  final  version  of 
the  job  inventory  is  prepared. 

Tile  inventories  are  then  mailed  to  unit  trade  test  officers 
(analogous  to  USAF  test  control  officers)  who  administer  tliem  under  standardized 
conditions  to  selected  groups  of  job  Incumbents  in  base  testing  rooms.  We 
should  note  litre  tliat  great  care  is  taken  to  select  representative  samples. 

In  the  larger  musterings  (N  800)  samples  between  50  and  75  percent  are  used 
while  U)0  pertent  sampling  is  the  goal  for  the  smaller  musterings. 

In  completing  the  inventories,  job  incumbents  provide  identification, 
background,  and  other  job  related  information.  Then  using  specially  designed 
machine-readable  answer  sheets,  they  respond  to  the  task  statements  by 
indicating  each  task  performed  and  the  relative  amount  of  time  spent  on  it. 
Finally,  ii.cunibeut  s add  tasks  not  listed  in  the  inventory.  Because  Lite  task 
response  answer  sheets  used  are  to  be  optically  scanned,  incumbents  wiiu--in 
their  additioudi  t.islsin  the  background  information  section,  not  on  the  ta.-^k 
answei  s'.ieets  tiiemselves. 

Completed  inventories  are  returned  to  us  at  Headquarters  Support 
Command  where  liu-v  ate  ehecked  for  errors.  Background  information  is 
forwarded  to  the  RAAF  EOF  Centre  for  keypunching  and  transfer  to  magnetic 
tape.  ITu  tasli  L'l'sp.  nses  go  to  a local  university  where  thev  are  vonvi'i'toU 
to  tape  by  an  ojUiial  scanner.  (By  the  way,  the  optical  staii  iiror  rate 

for  oui  laet  .i.iee  surveys  was  less  than  j percent.)  The  las'  response  t.ipe  is 
then  sent  to  t.ie  i\.A.AF  EUT  Centre  where,  by  means  of  subject  respondent  codes, 
the  two  tapes  ir.  merged  into  one  which  contains  the  complete  file  loi  e acli 
airman  in  .st(,n.,;.  ■ liie  data  are  then  re-edited  and  prepared  for  ce.  putir 
ana  i y - 


I hu-  -I  tie  major  advantages  of  the  job  inventory  .i()|)roach  to 
occupationai  i..il  ii  i tiiat  tu  results  are  comprehensive  and  easily 
quantified  u. hi  .til,  19/1).  To  fully  exploit  this  advantage,  we  ate  working 

with  the  RAM  l.oF  Ceotio  in  the  development  of  a catalogue  of  standard 
computet  ized  a . uj'.i  i i .ma  1 analysis  programs.  To  date,  we  have  available  twM 
programs  tor  ,'jipl  i.  at  ion  to  our  job  surveys  - group  job  descriptions,  and 
group  dittefiMUe  descriptions.  The  group  job  description  program  enables 
the  computer  1.'  c.  nerate  job  descriptions,  made  up  of  ta.sks  performed  and  the 
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average  percentage  of  time  spent  on  each  of  them  for  any  group  of  individuals 
where  the  groups  can  be  defined  in  terms  of  background  variables.  For 
example,  a job  description  can  be  computed  for  all  airmen  of  a given  rank,  or 
for  those  assigned  to  a specific  work  area,  or  for  those  who  have  been  on  the 
job  for  less  than  six  months.  Indeed,  a group  job  description  can  be  generate'^ 
in  terms  of  any  single  or  combination  of  background  variables.  Thus,  a grou) 
job  description  could  be  provided  of  all  corporal,  electrical  fitters,  between 
the  ages  of  20-25,  with  over  three  years  experience  in  the  mustering,  and  over 
one  year  in  their  present  position.  The  background  variables  and  possible 
combinations  on  which  groups  may  be  categorized  are  limited  only  by  the  job 
analyst's  imagination. 

Our  second  program  provides  comparisons  between  groups.  These  group 
difference  descriptions  effectively  point  out  the  difference  between  two 
groups  in  terms  of  the  emphasis  they  place  upon  various  tasks.  With  such 
descriptions,  it  is  possible  to  determine  how  the  time  spent  on  tasks  varies 
with  rank,  time  in  the  job,  time  in  service,  by  unit,  by  area  of  assignment, 
or  any  other  background  variable  completed  by  the  incumbents.  For  example, 
a comparison  by  rank  would  depict  how  the  job  changes  as  one  progresses  in 
rank;  or  which  tasks  are  performed  by  junior  members  and  which  by  more  senior 
ones.  A comparison  of  major  work  locations  would  indicate  which  tasks  are 
performed  by  airmen  assigned  to  flying  squadrons  as  opposed  to  those  which  are 
performed  by  aircraft  depot  members.  Greater  detail  concerning  the  specifics 
of  the  programs  can  be  found  in  the  writings  of  Archer  (1966);  Morsh  and 
Christal  (1966);  and  Christal  (1971). 

Those  of  you  familiar  with  the  job  inventory  approach  will 
immediately  recognize  that  the  above  described  programs  are  versions  of  two 
CODAP  (Comprehensive  Occupational  Data  Analysis  Programs)  programs  designed 
by  Dr  Raymond  E.  Christal  of  the  Air  Force  Human  Resources  Laboratory. 

While  the  RAAF  does  not  possess  any  of  the  other  CODAP  programs  at  present, 
it  is  studying  the  feasibility  and  cost-effectiveness  of  using  the  programs 
in  a relatively  small  force. 

of  RAA?'  Jooupcit-io>ul  Dati 

In  the  USAF,  the  basic  unit  of  the  airmen  occupational 
classification  structure  is  the  specialty.  Specialties  group  related  work 
positions  on  the  basis  of  knowledge,  education,  training,  experience,  and 
other  abilities  required  to  perform  them  Specialties  are  grouped  into 
larger  clusters  called  career  fields.  This  clustering  is  based  on  the 
similarity  of  the  specialties  and  their  transferability  of  skills  and 
knowledges  to  provide  optimum  utilization  and  career  development.  The  USAF 
presently  uses  some  46  career  fields  encompassing  approximately  235  specialties. 
Skill  distinction  and  progression  within  a career  field  is  indicated  by  a 
vertical  arrangement  of  specialties.  Thus  an  airman  moves  'up  the  career 
ladder'  from  apprentice  to  journeyman  to  supervisor  to  superintendent.  Each 
skill  level  within  a career  ladder  is  designated  as  a separate  snecialty. 

Within  the  RAAF,  airmen  are  assigned  into  25  primary  work  areas 
known  as  trade  groups  (similar  to  USAF  career  fields)  which  embrace  a total 
of  120  different  miisterings  (USAF  specialties).  Each  trade  group  is 
organized  along  as  functional  lines  as  possible.  For  example,  the  catering 
trade  group  contains  such  masterings  as  cook,  cook's  assistant,  steward, 
caterer,  and  butcher-  In  some  instances,  a mustering  is  used  as  a 'feeder' 
for  a more  specialized  mustering  within  the  same  trade  group  thus  establishing 
a career  plan  Thus,  a cook's  assistant,  by  further  training  and  testing. 
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could  progress  to  cook.  This  type  of  career  progress  is  somewhat  analogous  to 
the  USAF's  career  ladder  system.  It  differs  in  that  while  airmen  in  the  USAF 
are  expected  to  progress  in  terms  of  specialization,  RAAF  airmen  may  remain  in 
a mustering  with  little  upward  mobility.  The  scope  of  RAAF  work  activities, 
therefore,  is  accomplished  by  120  airmen  specialties  while  the  USAF  uses  over 
235. 


Because  of  the  smaller  number  of  muster ings,  you  might  expect  that 
RAAF  airmen  would  be  involved  in  a much  wider  variety  of  job  activities  than 
are  their  USAF  counterparts  working  in  similar  specialties.  Wliile  we  have  no 
USAF  data  on  comparable  specialties  available  at  the  present  time,  results  from 
our  first  two  occupational  surveys  (Sellman  & Crow,  1972;  Sellman  it  CrovT, 

1973)  suggest  some  support  for  this  notion.  Perhaps  some  examples  of  RAAF 
data  at  this  point  might  be  illustrative.  Tables  1 and  2 present  for  the 
total  survey  samples  and  the  major  operational  work  areas  the  extent  to  which 
work  time  is  spread  across  tasks.  This  is  done  in  terms  of  the  number  of 
tasks  accounting  for  10  percent,  25  percent,  50  percent,  75  percent,  90 
percent,  and  100  percent  work  time  as  tabulated  from  group  job  descriptions. 

It  can  be  seen  that  for  major  work  areas,  group  work  time  is  spread  across  a 
larger  number  of  tasks  for  the  operational  locations  such  as  aircraft  depots, 
flying  squadrons,  and  maintenance  squadrons  than  for  the  non  operational  areas 
such  as  the  training  schools,  command  headquarters,  and  the  Department  of  Air. 
The  number  of  tasks  accounting  for  work  time  can  also  be  considered  a measure 
of  job  specialization  - the  more  tasks  performed  the  lower  the  specialization, 
and  vice  versa.  For  example,  training  schools  and  DEPAIR  personnel  are 
highly  specialized  in  their  particular  jobs  because  they  spend  almost  all 
their  time  performing  a few  activities;  aircraft  depot  airmen,  on  the  other 
hand,  are  less  specialized  in  that  they  spend  their  working  liours  performing 
a large  number  of  tasks. 


TABLE  1 


NUMBER  OF  TASKS  ACCOUNTING  FOR  WORK  TI’fE 
ELECr.RICAL  ENGINEERING  MAJOR  ■’ORK  A.REAS 
AND  TOTAL  SURVEY  SAMPLE 


Number 

of  Tasks 

Account! 

ng  For 

10% 

25% 

50% 

75% 

90% 

100% 

of 

of 

of 

of 

of 

of 

Group 

N 

Time 

Time 

Time 

Time 

Time 

Time 

TRG  SCHOOL 

20 

1 

5 

19 

58 

126 

273 

DEPAIR/ CMD 

19 

1 

4 

13 

33 

61 

133 

MNT  SQN 

199 

19 

58 

151 

310 

507 

1146 

FLY  SQN 

146 

13 

40 

112 

255 

419 

899 

ACFT  DEPOT 

95 

12 

39 

119 

269 

446 

1120 

TOTAL  SAMPLE 

496 

17 

54 

147 

308 

491 

1148* 

*1148  Tasks 

Listed 

in  the  i 

Electrical 

Engineering  Job 

Inventory 

625 
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TABLE  2 


NUMBER  OF  TASKS  ACCOUNTING  FOR  WORK  TIME  OF 
INSTRUMENT  ENGINEERING  MAJOR  WORK 
AREAS,  ANT)  TOTAL  SURVEY  SAMPLE 


Number 

of  Tasks 

Accounting  For 

10% 

25% 

50% 

75% 

90% 

100% 

of 

of 

of 

of 

of 

of 

Group 

N 

Time 

Time 

Time 

Time 

Time 

T ime 

CMD/DEPAiR 

25 

2 

8 

24 

54 

89 

199 

TRG  SCHOOL 

25 

4 

14 

41 

101 

186 

353 

MNT  SQN 

209 

14 

48 

145 

211 

411 

1027 

FLY  SQN 

156 

12 

40 

108 

234 

391 

861 

ACFT  DEPOT 

88 

9 

31 

97 

237 

422 

922 

TOTAL  SURVEY 

515 

14 

47 

141 

299 

491 

1027* 

*1027  Tasks  Lisced  in  the  Instrument  Engineering  Job  Inventory 


Tables  3 and  4 present  a summary  of  the  average  percent  time  spent 
on  each  duty  for  the  major  work  areas  and  total  survey  samples  for  the 
electrical  and  instrument  engineering  musterings.  Inspection  of  these  tables 
plus  Tables  1 and  2 would  seem  to  confirm  the  idea  that  the  electrical  and 
instrument  engineering  musterings  are  extremely  heterogeneous  in  nature,  ie, 
airmen  are  involved  in  a wide  variety  of  work  tasks  spending  relatively  little 
time  on  any  of  them.  This  is  indicated  by  the  large  number  of  tasks 
accounting  loir  group  work  time  and  by  the  relatively  small  amounts  of  time 
devoted  to  non-supervisory  duties.  As  suggested  above,  these  findings  were 
not  unanticipated  since  RAAF  airmen  are  expected  to  be  skilled  in  all  work 
concerned  with  their  musterings.  Electrical  fitters,  for  example,  must  be 
proficient  with  all  electrical  systems  (including  ground  support  equipment) 
found  on  every  type  of  aircraft,  marine  craft,  and  road  transport  in  tlie  RAAF, 
while  instrument  fitters  work  with  all  aircraft  instruments,  instrument 
systems,  and  associated  equipment  including  gun  and  bombsights,  cameras, 
oxygen  systems,  and  cabin  pressurization  equipment. 

Now  let  us  look  at  some  example  data  which  will  appear  more  familiar 
to  you  since  it  closely  parallels  results  from  USAF  job  analyses.  Tables  5 
and  6 summarize  for  the  six  RAAF  enlisted  ranks  the  numbers  of  tasks  accounting 
for  10  percent,  25  percent,  50  percent,  75  percent,  90  percent,  and  100  percent 
of  the  work  time  for  the  electrical  and  instrument  engineering  musterings. 

(AC  - aircraftman,  LAC  - leading  aircraftman,  CPL  - corporal,  SGT  - sergeant, 
FSGT  - flight  sergeant,  and  WOFF  - warrant  officer). 


626 


/TABLE  i 


DUTY  OID/DEPAIR  TRG  SCHOOL  MNTSQN  ACFT  DEPOT  FLYSQN  TOTAL  SAMPLE 
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Test,  and  Simulator  Equipment  0.00  .03  .75  .34  .14  .20 


AVERAGE  PERCENT  TLME  SPENT  ON  DUTIES  BY  THE  INSTRUMENT  ENGINEERING 
MUSTERINGS  BY  MAJOR  WORK  AREA,  AND  THE  TOTAL  SAMPLE  (N=515) 
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Troubleshooting  Aircraft 

Systems  and  Components  00  .70  1.45  2.82 


DUTY  CMP/ DEPAIR  TRG  SCUOOL  MNTSQN  AC FT  DEPOT  FLYSQN  TOTAL  SAMPLE 
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!.  j;  ^LCOUHTlNi;  FOR  WORK  TLMF  OF 

KNi; INHERING  MUSTERINGS  BY  R/'G.'K 
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TABLE  6 

‘.'omrlR  of  tasks  accounting  for  work  time  of 

INSTRUMENr  ENGINEERING  MUSTERINGS  BY  R/YNK 


Number  of 
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. 1 ■■it  a;  ■ ,i  .;i  major  difference  between  i auk:-,  woe  Id  Dl  , ol 
■ I aupei  V i t,  ion  performed.  I'he  greater  the  rank,  Lite 
' i,.i  . t . ii  mi'll  performing  supervisorv  tasks,  .nut  greater 
I . t I in'  spent  on  supervision.  lionverse  1 v , as  l ank 
int  ot  t i..ie  and  the  percentage  performing  tasks  outside  ot 
less  i cnif  icant  factors  in  the  job  deseiiplion  ■'tf 
. . ii  . ,1  i>t  tanks  within  tlie  electrical  and  instrument 
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i'a>; i iieer  1 nft  miisterings  is  made,  tlie  above  described  phenomenon  does,  in  tact  , 
c,cur.  Tables  7 and  8 audicate  Lite  average  percent  time  spent  by  Liu  six 
e^li^.Led  ranks  across  Lite  two  musLerings  being  considered  for  each  of  the 
duties  in  the  job  inventory. 

In  certain  instances  it  is  importaiit  Lo  know  in  what  respect',  the 
work  performed  by  groups  is  dissimilar.  Such  information  may  clarify 
distinctions  by  ranks  or  between  groups  with  differing  experience  or  training. 
A computer  program,  the  group  difference  description,  has  been  devised  to 
compute  differences,  in  terms  of  members  performing  and  in  terms  of  average 
time  spent  on  each  task,  between  anv  two  groups  defined  by  background 
variables  (Morsli  and  Christal,  1966;  Christal,  1971).  All  tasks  in  the 
inventory  are  ordered  according  to  the  magnitude  of  the  differences  in 
percent  performing  between  the  two  groups  compared.  The  printout  hegin'. 
with  the  greatest  positive  difference,  proceeds  through  7.c-ro  ditlerence, 
and  ends  witii  the  greatest  negative  difference. 

After  careful  review  of  Table  7 and  8 information,  it  was  concjuiied 
liiaL  die  six  ranks  could  be  combined  into  two  meaningful  clusters  - ACs , , 

CTLs  into  a 'worker'  group  and  StiTs,  FSGTs,  and  U'OFKs  into  a 'supervisor' 
group.  Group  difference  descriptions  were  then  computed.  Tables  9 and  10 
provide  lists  of  the  24  tasks  that  most  significantly  discriminated  between 
the  workers  and  supervisors  for  the  electrical  and  instrument  engineering 
musterings,  and  the  differences  between  those  percentages  for  each  task. 

The  first  dozen  tasks  performed  by  'workers'  in  each  mustering  are  in  the 
servicing,  repairing  and  adjusting,  and  installing  and  replacing  duties. 
Supervisor  tasks,  on  the  other  hand,  fall  into  the  planning  and  organizing, 
ilirectlng  and  implementing,  and  inspecting  and  evaluating  duties.  These 
data  lead  to  the  somewhat  straightforward  interpretation  that  w.jrkers  art- 
engaged  in  the  technical  aspect  of  the  musterings  while  the  senior  N'COs  have 
involved  themselves  in  technical  management  and  supervision.  Thes''  art,  of 
course,  not  particularly  startling  results,  l,ui  i hev  dti  providt  additional 
1-.  i.ii'ii  I as  to  the  vaJidity  ol  Die  siir\'<  s-  i indiiu'-.. 


Now  let  us  conclude  hv  vny  oi  i<  i 1 d i ai  iiss  i iig  some  ot  our  on-going 
research  in  suppor..  of  the  occupational  analv-.i-  rrogram. 

One  advantage  of  occufiational  an.il.sia  ilmost  always  niont  i I'n.d  in 
Uie  literature  is  its  use  in  designing  ti  lining  systeiiis.  Yet  despite  the 
widespread  acknowledgement  of  the  efficaev  of  selecting  instructional 
objectives  from  up-to-date  job  information,  only  a few  researchers  (Smith, 
1964;  Smith,  1966;  Ammerman,  1966;  Ammerman  4 Melch'ng,  1966;  and  Hunter, 
Lyons,  MacC.islin,  Smith  & Wtigner,  1969)  have  applied  themselves  to  the  actual 
task  of  developing  procedures  f iir  translating  job  data  into  forms  specifically 
designed  lo  lu  I p individuals  responsible  lor  curriculum  definition.  ('sing 
Chrlstal's  (1970)  ideas  as  our  basic  guide,  we  are  presently  collecting  data 
on  five  of  the  right  factors  (probabilitv  oi  emergency  performance, 
probability  of  inadequate  performance,  consequences  of  inadequate  performance, 
still  per ishab  i 1 i t V , and  task  difficulty)  he  suggests  as  possible  criteria 
tor  selecting  tasks  for  inclusion  i ti  training  courses.  Upon  completion  of 
c'ur  aiialvMes,  we  hope  to  develop  an  algoritnm  whicli  can  be  used  by  training 
managers  to  convk’rt  occupati  .uial  information  into  tiaining  objectives  and 
I’urse  content.  Long  range  plan',  call  foi  t lu'  cuvelopmeiu  ol  computer 
programs  wliich  will  print  out  training  curricula  and  trade  standards  as  a 
i.-utine  part  ot  ol  ciipat  i ona  1 anale^,is. 
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Percent  of  Members  PerformlriR  in  Group 

SGTs,  FSGTs,  WOFFs  ACs,  L,\Cs . vJPLs  SGTs , FSGTs , WOFFs  Difference 


A second  project  on  wliich  we  are  currently  working  concerns  job 
satisfaction  within  the  RAAF . As  more  and  more  emphasis  is  placed  on 
improved  personnel  utilization  within  an  environment  of  defence  strength 
and  budget  reductions,  job  satisfaction  has  begun  to  take  on  an  increasing 
importance  within  the  military  services.  In  I96b,  the  US  Air  Force 
initiated  a program  to  evaluate  the  impact  of  work  assigned  on  job 
satisfaction,  felt  utilization  of  talent,  and  career  decisions  (Gould  i 
Christal,  197i;  Gould,  1972;  and  Christal,  1972).  Respondents  to  job 
inventories  use  a 7-polnt  scale  to  indicate  tlieir  job  interest  ranging 
from  'extremely  dull'  to  'extremely  interesting'  and  the  extent  to  wliich 
their  jobs  use  tiieir  talents  and  training  with  responses  from  'not  at  all' 
to  ' perfectly ' . 

Results  to  date  appear  very  promising  and  suggest  a powerful 
methodology  for  studying  the  attitudes  of  airmen  toward  their  work,  and 
for  identifying  possible  work  changes  within  masterings  wliich  may  improve 
job  satisfaction  and  thus  wc>rk  performance  (Stacy,  1972).  Consequently, 
we  are  now  capturing  job  satisfaction  data  as  a regular  part  of  our 
occupational  survey  procedures.  Unfortunately,  we  do  not  yet  have  any 
results  to  present  to  you;  however,  it  is  hoped  tiiat  by  next  year's 
conference  we  will  be  able  to  discuss  job  sat  i sf  ac  t ion  in  tlie  1G\AF  in 
some  detail. 


file  tuture  ot  occupational  analysis  in  the  Royal  Australian  .Air 
Force  looks  promising.  The  feasibility  of  the  job  inveiitorv  approach  has 
been  demonstrated,  and  we  are  constantly  striving  to  develop  innovations 
and  advances  in  our  inventory  construction  and  survey  administration 
techniques.  We  plan  an  expansion  of  our  occupational  survey  schedule,  and 
this,  combined  with  our  anticipated  research  results,  should  provide  the 
R.-VAF  with  job  information  which  will  lead  to  im.provements  in  tlie  management 
of  personnel  and  training  programs. 


REFEREN'CKS 


.■\mmerman,  H.I..  .' e ’ar”:-  ct  .y’  ■■  y’  ;•  ''}ii 

.■'O' 0 .'-iK'  r IIUMRRO  TR-b'b-3. 

.Alexandria,  V.A:  Human  Resources  Research  Office,  May  19hh. 


jAmmerman,  H.L.  and  .Melching,  W.H.  ' j i‘:.i 

jl.icsifijatioK  .y'  ob,T^^L‘f\’er.  HUMRRO  TR-6b-4 . 

Alexandria,  VA:  Human  Resources  Research  Office,  May  I9nh. 

Archer,  W.B.  t it  u.'H  of  fob  oVsef’;- ‘ y’.” * * ly  • ’ ■. 

l.i'i.  PRL-TR-66-12.  Lackland  AFB,  T.X : Personnel 

Research  Laboratory,  Aerospace  Medical  Division,  December  lybh. 


Christal.  R.E.  Implications  of  Air  Force  occupational  research  for 
curriculum  design.  Paper  presented  to  the  Minnesota  Research 
Co-ordinating  Unit  for  Vocational  Education  Seminar:  Process 

and  Techniques  ot  Vocational  Curriculum  Development.  University 
of  Minnesota,  .Vovember  1970. 


I 


I 


641 


, Clir  i si  a 1 


Christal,  R.E.  Combining  occupaiional  information  with  personnel  data 
to  resolve  manpower  management  problems.  Paper  presented  to  the 
NATO  Conference  on  Manpower  Planning  Models,  Cambridge,  England, 
September  1971. 

Christal,  R.E.  New  directions  in  the  Air  Force  occupational  research 

program.  Paper  presented  to  the  80th  Annual  Convention  of  the 

American  Psychological  Association,  Honolulu,  September  1972. 

'ould,  R.B.  HspjrtiJ  ’i  'KU'P-.'t  7>.  ^ ‘ 'c,'  7 Jt;  ♦ 'oi;  . 

ti'jlK'np  l%  .'l  c’  -7,;cc.  AFHKL-TR-72-7 . 

Lackland  AFB,  TX:  Personnel  Research  Division,  Air  Force  Human 

Resources  Laboratory,  January  1972. 

C.ould,  R-B.  and  Christal,  R.E.  Enlisted  joh  satisfaction  in  tlie  Air 
Force:  A study  at  the  task  level.  Paper  presented  to  the  13th 

Annual  Conference  of  tlie  Military  Testing  Association,  Washington 
DC,  September  1971. 

Hunter,  H.G.,  Lyons,  J.D.,  MacCaslin,  E.F.,  Smith,  R.G.,  and  Wagner,  H. 
•t  : i‘  lor-’Kj  • »:  • e 

: J.L  . HUMRRO  TR-69-9.  Alexandria,  VA;  Human 

Resources  Researcit  Office,  May  I9h9. 


Morsh , 

J.E. 

c./oh,  / 

Job  analysis  in  the  United 
1964,'  37,  7-17. 

Stated  Air 

Force  . 

Morsh , 

J.E. 

Collecting,  analysing,  and 

reporting;  : 

L nf  orma t i on 

describing  lobs  in  the  United  States  Air  Force.  Paper  presented 
to  the  77th  Annual  Convention  of  the  American  Psyc ho  log i ca i 
Association,  Washington  DC,  September  1969. 

Motaii,  .J.E.  and  Christal,  R.E.  y"  _•  ;■  ■. 

■ '-,J.  .'.■••■c  ■ • PRL-TR-bb-ig.  Lackland  AFb',  TX: 

I’ersonnel  Research  l.aboratorv.  Aerospace  lledicai  Division, 

October  1 966. 

Morsh,  J . F. . , Madden,  J.M.,  and  Christal,  R.E.  - 

ji'j'  r !‘j.  . WADD-TR-6l-i  i3.  Lackland  AFB,  IX: 
Personnel  Laboratory,  Wright  Air  Development  Division, 

February  1 96 1 . 

Sellnan,  W.S.  and  Crow,  R.R  ••ift’o*.’  ’t;  r 'i'  ‘ ■ T;'’  ■■ 

' ■.enr'i  up  ■ GT3-JA-72-3.  Melbourne,  Australia: 

HQ  Support  Command,  Royal  Australian  Air  Force,  December  1972. 

Sellman,  W.S.  and  Crow,  R.R.  of  ‘ i ■ 

>:p  • Kccy  ' ■ p GT3-JA-73-1  Melbourne,  Australia: 

HQ  Support  Command,  Royal  Australian  Air  Force,  April  1973. 

Smith,  R.G.  . ;'e ’•  ■ V'  ^ t'aini'ip  ./i\  cat'hrs . HUMRRO  Bulletin  II. 

Alexandria,  VA:  Human  Resources  Research  Office,  May  1964. 

Smith,  R-G.  ieapt'i  f 'Ki'tv^jtioruai  HUMRRO  TR-66-18. 

Alexandria,  VA:  Human  Resources  Research  Office,  November  1966. 

Stacy,  W.J.  iel:  ’ 7 t ■'  ->2  rw, 

■ p .p-  ‘ ivcer  .1.':  i- ■ AFHRL-TR-72-XX.  Lackland  AFB,  TX  : 

Personnel  Research  Division,  Air  Force  Human  Resources  Laboratory 
( In  Press)  . 


642 


A M F.  H I C A N ! N S 1 I r U T E S FOR  P i.  *.•  E A R C 


COLLECTION  OF  PERFORMANCE  DATA  AT  THE  JOB  TASK  LEVEL* 

CLIFFORD  P.  HAHN 

I want  to  briefly  describe  a procedure  AIR  is  using  in  a current 
study  for  the  Air  Force  to  develop  task-level  job  performance  criteria. 

The  basic  concept  is  one  of  mul ti -variate  indices  to  measure  job  per- 
formance. A major  distinguishing  characteri Stic  of  the  study  is  the  use 
of  40  or  more  task-level  performance  indices  per  job  as  opposed  to  a few 
global  measures. 

Perhaps  it  should  be  pointed  out  that  a fundamental  reason  for 
undertaking  the  study  was  not  to  come  up  with  measures  of  performance 
for  the  sake  of  determining  individual  worth,  but  rather  to  derive  measures 
that  could  be  used  to  validate  various  portions  of  existing  selection, 
classification,  assignment,  and  training  procedures.  The  ainnen  included 
in  the  study  had  initially  been  selected  with  aptitude  tests,  had  been 
assigned  to  gross  occupational  areas  through  the  use  of  clustered  aptitude 
scores  in  a classification  process,  and  had  been  trained  through  a combi- 
nation of  technical  schools  and  on-the-job  programs.  The  job  performance 
measured  during  this  study  of  journeyman-level  ainnen  is  thus  partly  a 
product  of  their  inherent  personal  capabilities  and  partly  a product  of 
the  selection,  classification,  assignment,  and  training  systems  they  went 
through.  The  basic  purpose  was  to  use  the  indices  developed  to  validate 
the  system  outputs.  Since  individual  job  perfoniiance  represented  a direct 
index  of  that  output,  it  is  hoped  that  it  can  be  used  for  these  purposes. 

As  indicated  earlier,  the  basic  concept  is  one  of  multiple  criteria 
for  evaluating  a total  job,  a major  job  segment,  or  with  appropriate  adapta- 
tions, major  dimensions  of  job  performance.  If  the  jobs  themselves  are  made 
up  of  a series  of  separate  identifiable  tasks,  it  seems  appropriate  to 
measure  success  on  those  jobs  by  measuring  success  on  the  discrete  tasks 
which  make  up  the  jobs.  It  is  assumed  that  within  any  given  occupational 
area  there  are  an  identifiable  number  of  job  tasks  which  can  be  used  to 

*Paper  for  presentation  at  the  1973  Military  Tcbting  Association  Convention, 
San  Antonio,  Texas  on  28  October  - 2 November  l973. 
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describe  all  jobs  within  that  area.  No  single  job  will  involve  all  of 
the  tasks,  yet  all  jobs  within  the  occupational  area  can  be  described  by 
some  combination  of  the  tasks  within  the  occupational  area.  If  procedures 
exist  for  obtaining  perfoniiance  indices  for  all  tasks,  then  any  given 
individual's  job  performance  can  he  appraised  only  on  those  tasks  included 
in  iiis  job.  The  Air  force  systian  ut  (ask  analysis  certainly  provides 
the  necessary  infonnation  on  a descriptive  basis,  and  we  are  attempting 
to  develop  relevant  performatice  infotmation  at  the  same  level.  Txisting 
teciiniques  for  clustering  tasks  into  duty  positions  and  occupational 
fic.'lds  should  be  ec|ually  appropriate  for  combining  task  perfoniance 
indices  into  clustc.-red  performance  imlicc'S  either  on  the  basis  of  job 
duties,  total  jobs,  or  job  dimensions.  If  this  basic  concept  of  jier- 
formance  evaluation  can  bo  effectively  implemented,  it  has  tremendous 
potential  for  validation  studies  of  all  kinds. 

The  most  comprehensive  imiileinentation  of  a task  level  model  would 
include  all  possible  tasks  and  tin's  would  be  theoretically  possible.  In 
most  practical  situations,  however,  and  certainly  in  our  study,  it  was 
necessary  to  pare  the  total  task  lists  down  to  a manageable  number.  A 
number  of  different  approaches  are  possible.  We  used  the  collective 
judgments  of  experienced  first  and  second  line  supervisors  in  an  itera- 
tive process  involving  first  groups  of  two  or  three,  then  ten,  and  then 
approximately  fifty.  How  many  l.isks  'an  be  included  depends  largely 
upon  the  resources  available  for  implementing  the  study  or  operational 
procedure . 

Implementation  of  the  task  level  model  in  its  most  comprehensive 
form  would  also  call  for  the  measurement  of  all  relevant  dimensions  of 
performance  for  each  job  task  selected.  This  too,  while  theoretically 
possible,  would  be  difficult  in  most  practical  situations.  It  is  impor- 
tant however,  to  recognize  that  there  are  several  different  dimensions  of 
task  performance  for  most  tasks.  Included  are  such  dimensions  as  quality, 
speed,  materials  and  facilities  used,  amount  of  outside  help  needed,  self- 
initiation and  others.  Again,  we  used  groups  of  experienced  first  and 
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second  line  supervisors  to  judge  which  of  the  dimensions  for  a given  task 
were  most  critical.  Critical  in  th'^  sense  that  they  provided  the  basis 
for  discriminating  betv;ecn  good  and  noor  performers.  This  process  in 
itself  was  illuminating.  For  cer'tain  task  dimensions,  the  supervisors 
were  able  to  identify  only  two  groups,  an  acceptable  performance  group 
and  an  unacceptable  group.  For  otlier  task  dirensions,  they  were  able  to 
identify  a high,  medium,  and  low  group,  and  for  a relatively  few  task 
dimensions  they  could  identify  a sujier  group,  an  average  expected  group, 
a minimally  acceptable  group,  and  an  unacceptable  group.  Quite  obviously, 
in  v/orking  with  tfie  task  level  approach,  the  scalability  of  task  dimensions 
must  be  i nves t i ga t i^d  . For  practical  reasons,  we  ended  up  using  a standard 
scale  for  all  task  dimensions,  but  it  will  be  interesting  to  see  if  the 
judgmentally  identified  groups  will  be  supported  by  the  empirical  data. 

The  process  for  generating  performance  indices  on  the  selectc  task 
dimensions  included  self-ratings,  supervisor  ratings  and  peer  ratings. 

While  the  liabilities  of  ratings  as  performance  indices  were  fully  recog- 
nized, in  this  case  the  requirement  for  developing  a system  that  would 
work  for  large  numbers  of  people  scattered  across  the  world  within  feasible 
cost  parameters  almost  dictated  the  use  of  ratings.  One  of  the  questions 
to  be  answered  by  our  effort  is  whether  or  not  ratings  at  the  task  level 
diminish  or  eliminate  some  of  the  previously  identified  problems  of  ratings 
as  measures  of  job  performance. 

Since  we  have  performance  ratings  from  job  incumbents,  supervisors, 
and  peers  we  will  be  able  to  determine  the  degree  to  which  the  three  groups 
agree  on  what  tasks  are  included  in  the  job  of  the  person  being  evaluated. 
Unless  there  is  good  agreement  between  the  groups,  rated  performance  is 
obviously  based  on  different  rather  than  identical  concepts  of  the  job. 

In  the  typical  global  rating  situation  one  has  no  data  at  all  on  this 
aspect  of  the  performance  rating  process.  Table  1 shows  some  data  from  a 
preliminary  analysis  of  46  cases  in  l.iie  Com  unications  Center  Operations. 
There  is  reasonably  good  agreement  although  far  from  perfect  and  it  varies 
from  task  to  task.  In  more  cases  both  the  supervisors  and  poors  under- 
include, althougli  there  are  most  exceptions  to  this  for  the  supervisors 
than  the  peers. 
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Analyses  of  the  extent  of  agreement  between  self-ratings  and  ratings 
of  supervisors  and  peers  as  well  as  between  two  supervisors  or  two  peers 
for  the  same  incumbent  and  between  supervisors  and  peers  will  be  made. 
Depending  upon  the  relationships  obtained,  it  may  be  possible  to  identify 
certain  tasks  or  groups  of  tasks  which  can  best  be  rated  only  by  super- 
visors, or  only  by  peers,  or  equally  by  both.  There  will  probably  be 
tasks  for  which  neither  supervisor  or  peer  ratings  are  really  appropriate. 
These  should  obviously  be  specifically  excluded  from  the  judgmental  base 
for  making  performance  ratings.  Other  tasks  might  be  excluded  on  the 
theoretical  basis  that  they  provide  no  performance  variance  and  therefore 
cannot  help  discriminate  between  different  levels  of  performance.  If 
one  were  using  the  performance  ratings  as  a measure  of  personal  worth, 
this  would  not  be  defensible  since  such  tasks  are  a part  of  the  total 
performance  picture.  If  the  performance  ratings  are  used  to  judge  the 
value  of  personnel  system  outputs  as  they  are  intended  to  in  this  case, 
the  elimination  of  such  non-variance  tasks  is  both  defensible  and  desir- 
able. 

Table  2 shows  some  data  from  a preliminary  analysis  of  performance 
ratings  for  47  cases  in  the  Coinnunications  Center  Operations  career 
field.  As  you  can  see,  the  range  of  correlations  is  considerable.  There 
appears  to  be  more  agreement  among  supervisors  and  peers  than  among  in- 
cumbents and  either  group,  with  one  exception. 

Another  aspect  of  the  task  level  approach  as  we  are  attempting  to 
develop  it  is  our  attempt  to  tease  out  from  the  levels  of  job  performance 
obtained  that  portion  which  is  due  to  skill  and  ability  factors  and  that 
portion  due  to  motivational  factors.  In  order  to  obtain  data  bearing  on 
this  question,  incumbents  were  asked  to  rate  both  the  intensity  and 
direction  of  six  factors  for  each  task.  These  factors  included  the  nature 
of  the  actual  work,  the  importance  of  the  task,  the  match  between  task 
difficulty  and  incumbent  capabi 1 i ties , the  opportunity  for  praise  and 
recognition,  the  amount  and  kind  of  supervision,  and  personal  relations. 
Incumbents  were  also  provided  the  opportunity  for  write-in  factors  of  their 
own  choosing. 
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Table  3 shows  some  data  from  a preliminary  analysis  of  the  motivational 
ratings  for  43  cases  from  the  Conmuni cations  Center  Operations  career 
field. 

A review  of  Table  3 reveals  the  relatively  small  per  cent  of  all 
possible  tasks  ranked  as  being  done  and  having  an  appreciable  effect  on 
motivational  level  for  a given  duty  position.  This  would  tend  to  indicate 
that  there  may  be  a great  deal  of  masking  of  results  when  overall  moti- 
vational ratings  across  fairly  broad  occupational  groups  are  obtained. 

It  appears  that  self-judged  importance  of  the  task  is  the  predominant 
motivational  factor  at  the  task  level,  and  that  it  is  largely  positive  in 
nature.  On  the  other  hand,  the  opportunity  for  praise  and  recognition 
appeared  to  have  much  less  potency  and  it  was  about  as  likely  to  be 
positive  as  negative  in  nature,  four  of  the  six  factors  tended  to  be 
largely  positive  in  nature. 

Supervisors  and  peers,  after  having  ranked  the  performance  level  of 
an  incumbent  on  a task  dimension,  were  asked  to  make  a judgment  regarding 
what  it  would  take  for  the  incumbent  to  reach  his  full  performance  potential 
on  that  task--either  the  acquisition  of  additional  skills  and  knowledge, 
the  raising  of  his  motivational  level,  or  combinations  of  both.  A five- 
point  scale  was  used  for  this.  Analyses  of  these  data  should  give  an 
indication  of  the  types  of  tasks  for  which  performance  levels  are  dictated 
largely  by  skill  and  ability  factors,  those  which  are  largely  dictated  by 
desire  to  do  a good  job,  and  those  which  are  an  admixture  of  both.  Skill 
determined  tasks  are  normally  thought  of  as  somewhat  unique,  while  motiva- 
tionally determined  tasks  could  probably  be  expected  to  be  task-general, 
but  person-unique.  Whether  or  not  these  expectations  will  be  shown  to 
be  empirically  true  remains  to  be  seen. 

If  analyses  of  the  data  collected  indicate  that  we  have  produced 
the  kinds  of  indices  that  we  had  hoped  we  would,  the  major  impact  on  the 
potential  for  better  utilization  of  human  resources  will  derive  from  the 
fact  that  there  will  be  an  abundance  of  data  for  many  different  validity 
studies.  Each  segment  of  the  personnel  management  system,  e.g.,  selection, 
classification,  assignment,  and  training,  can  be  differentially  subjected 
to  more  meaningful  validation  studies. 
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Table  1 

Agreement  Between  Incumbents,  Supervisors,  and  Peers  on  Tasks 
Included  in  Incumbents  Job 
(N-46  - 29150  AFSC) 


Supervisors  Peer 
Inclusion  Inclusion 
as  X of  as  X of 


Task 

X Incumbents 

« Supervisors 

Peer 

I ncumbent 

Incumbent 

Dimension 

I ncl  us  ion  __ 

Inclusion 

Indus  ion 

Inclusion  _ 

Inclusion 

1 

.80 

.73 

.64 

.91 

.80 

2A 

.50 

.48 

.38 

.96 

.76 

2B 

.52 

.50 

.41 

.96 

.79 

3 

.70 

.71 

.59 

1.01 

.34 

4A 

.34 

.36 

.38 

1 .06 

1.12 

4B 

.34 

.34 

.37 

1.00 

1.09 

5 

.67 

.71 

.58 

1 .06 

.87 

6 

.63 

.70 

.55 

1.11 

.87 

7 

.76 

.68 

.55 

.89 

.72 

8A 

.72 

.63 

.56 

.88 

.78 

8B 

.85 

.70 

.56 

.82 

. 66 

8C 

.72 

.61 

.56 

.85 

.78 

9 

.78 

.77 

.65 

.99 

.83 

10 

.67 

.64 

.65 

.96 

.97 

llA 

.76 

.70 

.59 

.92 

.78 

IIB 

.70 

.66 

.55 

.94 

.79 

12 

.57 

.50 

.41 

.88 

.72 

13 

.78 

.71 

.60 

.91 

.77 

14 

.63 

.70 

.54 

1 .11 

.86 

15 

.74 

.68 

.51 

.92 

.69 

16 

.74 

.70 

.55 

.95 

.74 

17 

.67 

.61 

.50 

.91 

.75 

18 

.67 

.50 

.47 

.75 

.70 

19 

.61 

.45 

.44 

.74 

.72 

20 

.57 

.59 

.47 

1.04 

.82 

21 

.80 

.64 

.55 

.80 

.69 

22 

.76 

.73 

.49 

.96 

.64 

23A 

.61 

.52 

.41 

.35 

.67 

23B 

.48 

.43 

.41 

.90 

.85 

24 

.52 

.50 

.44 

.96 

.85 

25 

.43 

.14 

.37 

.33 

.86 

26 

.37 

.11 

.29 

.30 

.78 

27 

.46 

.29 

.29 

.63 

.63 

28 

.43 

.29 

.28 

.67 

.65 

29 

.30 

.21 

.19 

.70 

.63 

30 

.33 

.18 

.18 

.55 

.55 

31 

.41 

.27 

.27 

. 6b 

.66 
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(Continued) 


Task 

I ncuiiiben 

Diii;onsion 

Inclusion 

32 

.37 

33 

.41 

34 

.48 

35 

.54 

36 

.50 

37 

.52 

38 

.46 

39 

.48 

40 

.57 

41 

.61 

42 

.43 

43A 

.70 

43B 

.67 

44 

.67 

a 

/o 


Supervisors 

Peer 

Inclusion  _ 

Inclusion 

.29 

.26 

.25 

.24 

.45 

.40 

.55 

.35 

.45 

.40 

.36 

.31 

.38 

.35 

.45 

.40 

.46 

.38 

.50 

.42 

.32 

.24 

.64 

.45 

.61 

.42 

.57 

.36 

Supervisors 

Peer 

Indus  ion 

Inclusion 

as  % of 

as  °'o  of 

Incuirrbent 

Incumbent 

Inclusiofi 

I nd  us  i on 

.78 

.70 

.61 

.59 

.94 

.83 

1.02 

.65 

.90 

.80 

.69 

.60 

.83 

.76 

.94 

.83 

.81 

.67 

.82 

.69 

.74 

.56 

.91 

.64 

.91 

.63 

.85 

.54 
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Table  2 

Correlations  Betv^een  Performance  Ratings 
by  Different  Types  of  Raters 
29150  - Communications  Center  Operations 
(Based  on  47  Cases  with  Ratings  Aggregated  Across  all  Tasks'  Rates) 


Supervisor 

Supervi sor 

Peer 

Peer 

1 

. ^ 2_ 

1 

2 _ 

Incumbent 

.29 

.93 

.27 

.09 

N=  47 

N=  10 

fj=  43 

N=  32 

n-999 

n=235 

ti-305 

n 603 

Supervisor  1 

.50 

.34 

.34 

N=  11 

43 

N-  31 

n=234 

n - /92 

n-568 

Supervisor  2 

.27 

N=  9 
11=209 

Peer  1 

.51 

N-  32 
n=482 
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Table  3 

Distribution  of  Incumbents'  Motivational  Rankings 
(Based  on  43  cases  each  having  potential  rankings  for  44  tasks) 
(Most  Important  = 1 Least  Important  = 6) 


Motivational 

Factor 

Mean 
Ranking 
All  Tasks 
Included 

Per  Cent 
of  all 
Tasks 
Ranked 

Mean 

Niii;;ber 

Qf 

T asks 
Ranked 

Mean 
Ranking 
of  Tasks 
Ranked 

Direction  of  Motivational 
Influences  - Per  Cent 
of  all  Tasks  Ranked 

Positive  Negative  P/N 

Importance  of 
Task 

.58 

42 

793 

1.39 

81 

8 n 

Nature  of 

Actual  Work 

.57 

30 

570 

1 .90 

74 

15  11 

Match  Between 
Task  Di ffi - 
cul ty  and 
Incumbent 
Capabi 1 i ty 

.36 

15 

274 

2.50 

72 

19  9 

Amount  and 
Quality  of 
Supervision 

.37 

15 

282 

2.53 

72 

14  14 

Personal 

Relations 

.29 

10 

197 

2.85 

55 

6 39 

Opportunity 
for  Praise 
and  Recogni- 
tion 

.29 

9 

170 

3.27 

40 

33  27 
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Table  4 

Mean  Motivational  Ranks  by  Task  and  Factor 


Task 

Number 

Importance 

of 

Task 

Nature 

of 

Work 

Match  Between 
Di fficul ty 
and 

Capabi 1 i ty 

Amount  and 
Quality  and 
Supervision 

Personal 
Relati ons 

Opportuni ties 
for 

Praise  and 
Recognition 

1 

1 .30 

1 .95 

2.50 

2.71 

2.50 

3.40 

2 

1.29 

2.70 

2.50 

1.90 

2.50 

3.33 

3 

1 .29 

1 .93 

2.71 

1 .80 

2.67 

2.50 

4 

1.44 

1.75 

3.33 

2.40 

3.50 

4.00 

5 

1.22 

2.07 

2.60 

3.00 

3.38 

3.00 

6 

1.20 

1.71 

3.00 

3.50 

2.33 

4.00 

7 

1.31 

1 .94 

1.88 

2.40 

1 .67 

2.00 

8 

1 .46 

1 .81 

2.08 

3.20 

3.00 

3.75 

17 

1 .25 

1.77 

2.83 

2.29 

4.00 

4.00 

18 

1.47 

1.79 

3.00 

2.17 

3.67 

2.67 

25 

1 .29 

2.08 

3.29 

2.13 

3.25 

3.67 

26 

1.33 

1 .82 

2.33 

4.00 

1.67 

1 .50 

27 

1 .42 

2.43 

2.50 

2.00 

2.40 

3.14 

28 

1.73 

1.88 

2.67 

2.00 

1.25 

2.67 

33 

1 .63 

1 .63 

2.75 

1 .60 

3.67 

3.33 

34 

1.29 

2.00 

2.50 

3.44 

3.00 

3.33 

35 

1.40 

2.17 

2.63 

2.22 

2.40 

2.67 

40 

1.29 

1 .60 

1 .67 

2.00 

2.33 

1 .00 

41 

1.37 

1.71 

2.00 

2.86 

3.67 

3.50 

44 

1 .26 

1.83 

3.00 

2.00 

4.00 

5.00 

Mean 

Across 

Tasks 

1.36 

1.93 

2.59 

2.48 

2.84 

3.12 

Range 

1.20-1.73  1 

.60-2.70 

1.67-3.33  1 

.60-3.50 

1.25-4.00 

1 .00-5.00 

*S.D. 

.13 

.27 

.44 

.65 

.80 

.93 

*S.D.  = Standard  Deviation 
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Table  5 

Motivational  Direction  offlature  of  Work  for  20  Tasks' 


Task 

Per  Cent 

of  Rankings 

Made 

Number  of  Ranks 

Number 

Positive 

Negative 

P/N 

Made 

1 

62 

10 

28 

21 

2 

70 

0 

30 

10 

3 

60 

33 

7 

15 

4 

75 

25 

0 

4 

5 

71 

7 

22 

14 

6 

83 

8 

8 

12 

7 

89 

11 

0 

18 

8 

81 

14 

5 

21 

17 

54 

38 

8 

13 

18 

79 

14 

7 

14 

25 

75 

17 

8 

12 

26 

64 

18 

18 

11 

27 

43 

57 

0 

7 

28 

63 

25 

12 

8 

33 

38 

50 

12 

8 

34 

93 

0 

7 

15 

35 

100 

0 

0 

12 

40 

40 

30 

30 

10 

41 

57 

21 

22 

14 

44 

75 

17 

8 

12 

Mean  Across 

Tasks 

68.6 

19.8 

11.6 

Range 

38-100 

0-57 

0-30 

Standard 

Deviation 

17.1 

15.7 

10.0 
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Table  6 

Motivational  Direction  of  'Importance  of  Task 


Task 

Per 

Cent  of  Raxkiln^s 

Made^ 

Numbex 

Po^i  t.i.ye 

Negative 

P/N 

1 

81 

12 

7 

2 

79 

0 

21 

3 

82 

12 

6 

4 

78 

0 

22 

5 

94 

0 

6 

6 

80 

10 

10 

7 

88 

4 

8 

8 

81 

0 

19 

17 

88 

6 

6 

18 

59 

35 

6 

25 

81 

9 

10 

26 

83  . 

8 

9 

27 

58 

17 

25 

28 

55 

9 

36 

33 

75 

12 

13 

34 

88 

12 

0 

35 

95 

0 

5 

40 

82 

12 

6 

41 

80 

16 

4 

44 

84 

5 

11 

Mean  Across  _ 

Tasks 

79.6 

9.0 

M .b 

Range 

55-95 

0-35 

0-36 

Standard  „ „ 

Deviation 

10.8 

8.2 

H.a 

Number  of  Ranks 
_ Made  

26 

14 

17 
9 

18 
20 
26 
22 
16 
17 
?1 
12 
12 
11 

8 

17 

20 

17 

19 

19 
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Table  7 

Motivational  Direction  of'Match  Between  Task  Difficulty 
and  Incumbent  Capability' 


Task 

Per 

Cent  of  Rankinos 

Made 

Number  of  Ranks 

Number 

Positive 

Negative 

P/N 

Made 

1 

83 

17 

0 

6 

2 

72 

12 

13 

8 

3 

86 

14 

0 

7 

4 

100 

0 

0 

3 

5 

90 

0 

10 

10 

6 

75 

25 

0 

4 

7 

75 

25 

0 

8 

8 

67 

8 

25 

12 

17 

83 

17 

0 

6 

18 

20 

60 

20 

5 

25 

57 

14 

29 

7 

26 

50 

33 

17 

6 

27 

33  • 

50 

17 

6 

28 

33 

67 

0 

3 

33 

50 

50 

0 

4 

34 

100 

0 

0 

8 

35 

63 

25 

12 

8 

40 

33 

33 

34 

3 

41 

75 

25 

0 

4 

44 

83 

0 

17 

6 

Mean  Across 

Tasks 

66.6 

23.8 

9.7 

Range 

33-100 

0-67 

0-34 

Standard 

Deviation 

23.5 

20.1 

11.3 
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Table  8 

Motivational  Direction  of'Ainount  and  Quality  of  Supervision' 


Task 

Per 

Cent  of  Rankings  Made 

Number  of  Ranks 

Number 

Positive 

Negative 

P/N 

Made 

1 

57 

29 

14 

7 

2 

80 

10 

10 

10 

3 

60 

40 

0 

5 

4 

70 

10 

20 

10 

5 

67 

0 

33 

9 

6 

100 

0 

0 

2 

7 

80 

0 

20 

5 

8 

60 

0 

40 

5 

17 

43 

43 

14 

7 

18 

33 

50 

17 

6 

25 

63 

25 

12 

8 

26 

100 

0 

0 

2 

27 

33 

50 

17 

6 

28 

40 

40 

20 

5 

33 

80 

20 

0 

5 

34 

78 

11 

11 

9 

35 

78 

0 

22 

9 

40 

75 

25 

0 

4 

41 

100 

0 

0 

7 

44 

100 

0 

0 

8 

Mean  Across 

Tasks 

69.9 

17.7 

12.5 

Range 

33-100 

0-50 

0-40 

Standard 

Deviation 

21.6 

18.7 

11.7 
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Table  9 

Motivational  Direction  of 'Personal  Relations' 

Task  Per  Ci  fit  of  Rankiinjs  ’"ide  ’Juii'ber  of  Ranks 


Nuiiiber 

Posi ti ve 

fi  ''i)ati  ve 

P/N 

Made 

1 

33 

17 

50 

6 

2 

25 

0 

75 

8 

3 

67 

0 

33 

3 

4 

67 

0 

33 

6 

5 

63 

12 

25 

8 

6 

67 

0 

33 

3 

7 

83 

0 

17 

6 

8 

71 

0 

29 

7 

17 

33 

0 

67 

3 

18 

33 

33 

34 

6 

25 

75 

0 

25 

4 

26 

67 

0 

33 

3 

27 

60 

20 

20 

5 

28 

75 

0 

25 

4 

33 

67 

0 

33 

3 

34 

50 

0 

50 

4 

35 

60 

0 

40 

5 

40 

33 

0 

67 

3 

41 

67 

0 

33 

3 

44 

75 

0 

25 

4 

Mean  Across 

Tasks 

58.6 

4.1 

37.4 

Range 

25-83 

0-33 

17-75 

Standard 

Deviation 

17.5 

9.1 

16.3 
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Table  10 

Motivational  Direction  of Opportuni ty  for  Praise  and  Recognition' 


Task 

Per 

Cent  of  Rankings  Made 

Number  of  Ranks 

Number 

Positive 

Negative 

P/N 

Made 

1 

40 

20 

40 

5 

2 

67 

0 

33 

3 

3 

50 

25 

25 

4 

4 

33 

33 

34 

3 

5 

67 

33 

0 

3 

6 

100 

0 

0 

1 

7 

50 

50 

0 

2 

8 

50 

25 

25 

4 

17 

100 

0 

0 

3 

18 

33 

50 

17 

6 

25 

33 

33 

34 

3 

26 

0 

100 

0 

2 

27 

0 

71 

29 

7 

28 

20 

60 

20 

5 

33 

0 

33 

67 

3 

34 

67 

33 

0 

3 

35 

67 

33 

0 

3 

40 

0 

100 

0 

1 

41 

0 

50 

50 

4 

44 

25 

0 

75 

4 

Mean  Across 

Tasks 

40.1 

37.5 

22.5 

Range 

0-100 

0-100 

0-75 

Standard 

Deviation 

31  .8 

29.3 

23.3 
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34UAJRCU  l.yJlD:-?.  BHIAt:  N RJHI'.Y  T-kF 
DEFENCE  rELI^Q.V.  DJaiCr:  STO,  lES  UlilT 
BEUCLI-UCH  PUCE.  ?JIlIVlE3nT  CF 
EJINHJRGH.  CCOTIAyp 


Abstract 

The  concept  of  coe^iitive  complexity,  its  relationship  to  personality 
VEiriables,  and  in  particular  its  association  with  the  perfortr.unce  of  leadership 
functions  13  briefly  reviewed.  Four  postulates  are  advanced  and  exaunined: 

Prevailing  contingency  theories  may,  by  over-emphasising  the  situational 
deteminants  of  leadershl[  , tend  to  obscure  peiTianent  sources  of  promlntTic.e 
and  influence  operating  in  a wide  range  of  situations. 

Intelligence,  conventionally  measured  as  a convergent  ability,  does  broadly 
predict  potential  occupational  status j but 

Intelligence,  above  certain  cut-cffs  and  nithlj.  fairly  broad  bands,  neither 
predicts  excellence,  nor  does  it  aid  vocational  choice,  nor  (especially) 
does  it  distinguish  potential  leaders;  but 

Cognitive  complexity,  and  specifically  that  element  which  is  defined  as  the 
ability  to  differentiate,  is  a pem!uier;tly  list  InguislJng  characteristic  of 
potential  leaders. 

The  construction  of  an  experiB.ental  differentiation  test  or  exercise  ia  described 
and  discussed  in  detail  and  it  is  concluded  that  objective  measures  of  cognitive 
complexity  may  usefully  be  added  to  leader  selection  batteries  and  appraisal  training 
programmes  so  as  to  enhance  the  utilisation  of  human  resources  in  conditions  of 
growing  demands. 

Intentions.  Assumptions  and  Plan 

I want,  if  I may,  to  talk  briefly  about  intelligence  and  leaderslilp:  not  to 
define  nor  really  to  explain  either,  since  far  better  men  tlian  I have  stumbled  on 
that  rock,  but  to  investigate  with  you  a small  part  of  the  eissociatian  of  these  two 
variables  eind  to  explore  an  idea  that  objective  measurement  of  a particular  facet  of 
the  former  may  usefully  predict  performance  in  some  functicnal  aspects  of  tht.  latter, 

I start  with  the  belief  that  since  the  extensive  and  well  documented  controversy 
over  the  relative  merits  of  clinical  and  statistical  prediction  (l),  there  Is,  to 
put  it  at  its  mildest,  doubt  tlirown  on  the  reliability  and  validity  of  subjective 
evaluations  made  in  selection.  Any  objective  measure,  therefore  wbiic.h  usefully 
oorrei'’tes  with  some  aspect  of  leadership  ;)C-rformanC'0  ia  to  be  welcomed. 

1.  dee  for  exeunple  Cliniced  Va  statistical  piedlction*  lieehl  1954,  Dniversity  of 
Minnesota  Press;  or  Measurement  and  prediction,  clinical  and  statistical,  lawjer, 
Psychological  Bulletin  Vol  66,  No  3,  dept  56. 

659 


Not  that  everycaie,  of  course,  subscribes  to  this  view  of  appraisal.  In  a 
recent  Sunday  Tlaies  article  on  selecting  a pony  for  a child,  readers  were  advised 
to  look  for  large  Intelligent  eyes,  wfdlst  the  guidance  given  in  this  matter  of 
world  Importance  included  the  following: 

(and  I quote) 

"Kind  ponies  have  kindly  eyes,  piggy  little  eyes  reveal  piggy  little  natures"  (2) 

Nor  was  objectivity  always  attainable  or  valued  by  professionals.  You  will 
recall  that  for  Spearman  when  writing  early  in  his  career  at  the  turn  of  the  century, 
intelligence  was  just  four  kinds  of  mental  activity  refreshingly  simple  to 
conceptualise  and  even  easier  tc  assess.  Two  of  these  you  will  remember  were: 

"General  Impression  produced  upon  other  people,  ie  bright,  average,  or  dull 
and 

which  was  established  by  asking  the  oldest  pupil  to  say  who  had 
the  most  'sharpness  and  commonsense  out  of  school',  and  who  next,  and  so  on. 

And  finally,  ty  way  of  a reliability  check,  the  supreme  test,  "a  similar  list 
was  obtained  from  the  rector's  wife  who  also  had  always  lived  in  this  village, 
but  her  graduation  is  unfortunately  incomplete  and  therefore  unusable " (3) 

At  least  the  minister's  wife  had  always  lived  in  the  village  - an  experience  few 
personnel  selection  officers  can  call  on. 

There  is  now  of  course  a multiplicity  of  intelligence  and  personality  tests, 
motivation  measures,  and  so  on  but  there  is  still  a need  to  find  simple  objective 
measui-es  dii-ectly  germane  to  the  identification  of  leadership  potential. 

I shall  try  therefore  to  describe  ny  own  first  faltering  steps  in  this  direction 
towards  one,  lock  in  detail  at  construction  of  the  instrument,  and  finally  discuss 
its  use  in  experimental  suid  training  situation.  1 would  welcome  comment  upon  this 
line  of  investigation,  but  first  let  me  set  the  scene  with  some  thoughts  on  concepts 
of  mind  and  of  leadership  which  it  seems  to  me  are  not  without  relevance. 


2.  Daddy,  dadcfy  can  I please  have  a pony.  Silver  and  Bousefield,  Sunday  Times, 

6 Apr  1973. 

3.  General  Intelligence  objectively  determined  and  measured,  Spearman  C,  American 
Journal  of  Psychology,  Vol  115,  1904. 
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The  ocncijpt  of  laind  has  vl-ways  been  ioiuitiuii^;  of  i . .. 

akin  to  'seal',  it  baffles  the  intellect,  eviies  logic,  eludes  scap  r«>hen:' i ;r.. 
i'h^'Siological  knowledge  of  its  fimdas'.ent'U  '.xirts  uid  toeir  fa;wti,.tun ; , i;  •/■  ■ ... 
Jomjarativ-''  infancy,  r'^ii'.es  littlj  vn ; i.c  ’.ut  ft  v case.;  to  the  a ,p.i  an:  if 

'syoliology,  Aiether  “’hese  ti.eoretical  o:nstru.-ts  oe  deri /ed  cy  s-nrlc  '/•> 

.ntrospecticn,  or  fron  :aore  sophistioated  f u;tor;  ,1  inaiyses,  or  inileei  from  tnc 
theory  of  radio  tranamittera  and  receivers,  they  do  .not  physically  fit.  The 
neurological  structui'e  of  mind  bears  reaembl.ance  neither  to  '/onion's  helrarcly  of 
abilities  (4)  nor  to  Guilford's  cube  (5).  'Viulst  the  literature  of  psychology  does 
strive  after  a tidy,  neat  and  comprehensible  system  of  classificaticn  uvJ  order, 
even  the  boundaries  beteeen  its  several  branches  lack  precisicn  and  are  not 
luiambiguous  but  the  line  of  demarcation  between  thought  euid  personality  in  piarticulax, 

between  ability  and  traits,  between  cogiitive  style  and  intelligence  is  so  blurred 

and  intangible  as  to  be  non-existent, 

liven  intelligence,  more  often  measured,  tested,  analysed,  diagnosed,  explained, 
written  about  than  any  other  single  human  attribute,  is  yet  but  an  operational 
definition.  We  know  not  what  it  reeolly  is.  In  the  linguistic  and  conceptual  chaos 
of  the  psychology  of  personality  and  oiind  it  is  a brave  sind  foolish  man  who  darea 

suggest  he  understauids.  So  also  is  it  with  the  equally  baffling  matter  of  leadership 

style  and  potential  - charisma  and  capability.  But  let  us  try  auii  uni'a'/el  some  of 
this. 

Like  Ceasar's  Gaul,  the  long  history  of  leadership  studies  falls  naturally  into 
three  parts  which  in  this  company  hardly  need  mention.  In  the  first,  biography  and 
autobiography,  reports,  news  items,  letters,  forna.1  and  informal  accounts  of  great 
men  were  scoured  for  the  personality  attributes  of  charismatic  greatness.  Personal 
character  qualities  were  thought  to  be  the  defining  characteristics  of  the  leader 
and  Indeed  there  still  are  appraisal  systems  in  which  this  somewhat  impracticable 
premise  plays  a dominant  part.  In  the  second  phase,  emphasis  shifted  from  what  the 
leader  is  to  what  he  does.  Small  group  theory  held  sway  and  since  ary  sociologist 
half  worthy  of  the  name  can  always  rustle  up  a few  subjects  for  a quick  rt  seer  cl  , 

there  mu.st  nov;  be  i hLlluui'i  b'lllon  studies  of  t:ie  factors  affecting  t.-ie  cnocgeuoe 

of  a liad-T  In  a iiri.11  Lni'n'ii'.  gv  lup.  r-f, -.jr.  groups,  triadi,  fon.'s  xil  t”i;, 

billions  j;’  trv*n,  ml  ; ..sc  ? tno  / 1 1 \b  ly  j'  -.i"  fiiilinga  In  tnese  fui  •’.iuii  iuilies 

4.  The  structu.ce  of  huaai  r.'i.U:.L‘S,  /-.•rrion  - 4,  letrujen  14‘V). 

5.  Three  faces  of  Intellect,  Gullfori  I P,  A,ijeri''.am  Psc'chologist  Vol  U,  l.fV* 
and  others  by  the  sane  ait  lor. 
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contradi:;tei  t.it?  otSiers  thor'S  developei  th‘>  t iird  pha-TP  whi'h  iVij'  b-  called, 

"it  all  dei»!nd3  on",  or,  "leaderaiiip  Is  aitnat  Lonally  deteminPd" , a:3  for  ex.uiplp 
in  J M Barrip's  tale  of  tiie  Adnirabl^  Crichton,  or  as  »as  hopp  fonually  stated  bj- 
iVarr: 

"This  leads  to  tne  paradox  that  p3y'--'holo;V  C*r!i*'h  usiaCly  seen  aa  a stody 

of  people)  Must  tarn  more  to  stjly  en'/i ri>mi‘*nts  and  oi".  ritio*i«  if  its 

developmental  moiaentim  i ; not  to  b-'  loit".  (o) 

Foremost  in  t.ii-:  fi-=ld  of  continy'.iy  I’leor^es  »i-,  u.  1 ;■  ...  i 

Fred  E Fiedler,  diuall  in  atatare  ait  i i not  uilix"  an  i;-.-:.i  t-T.-; -r  - i.t 

•Vila  a personality  t-j  .m  itoii  - tais  ^lant  in  "llect  fr-j.m  the  .iiiv--rjit.  • i . .*h  ■'t  n 
reduced  the  ciaioa  .of  coafl l'’tiny  iea.i-ir.ship  st.iies  t-j  a s-iitli.  . ; 

explained  t!'..-  apparent  inronsi  iten.-.i-'s  in  his  results  in  1*t-j  .•  it.,  il- 

First  V-iriabl^  yon  recall  was,  ’the  fav-ju;' iMen.’ss  of  t.  ^ . • aier* 

Iposition  po'ver,  task  atpieture,  relat  ion  i!;!  ps  , Jec-ml  v-ar.a  . , ' t i*  jiheiit  . o.. 

o'"  the  l=ad.’r'  (task  orientated  or  rel  ation-shi  i ori-'ni  atel  . .1.1.  »)  rsn.p  style 

was  to  be  .xeasnnni  by  an  ASO  score  ithe  assnnei  sinuiarit.  n i ..  . • . . »ot 

P erred  'o-worker  a;id  most  pi>!ferc»,.i  ’i'.*  jrkrr  or  LF.  a.  1 '.tPC  , 

But  you  know  and  I know  th.it,  nolwlthstan-Ln^  tiiat  expl'Uiat  ot.  1. . i a.i.  .ttin." 
tne  impoctan(;e  of  situational  factors,  th-'re  ar**  .some  men  w!io  more  often  tnan  most 
fill  the  leadership  po-sition  in  almost  any  i^oup  an  i almost  re^’anliejs  ot  place  an-l 
time,  whilst  tnere  are  others  who  never  do,  no-  ever  will,  Tni'r--  is  it  seems  therefo"e 
a need  to  resist  the  over-emnhn.sis  of  s i f n’’ ■ on  al  deteminancr/  an-i  still  a need  to 
identlf^’  the  p.>rni!uient  sources  of  infiuene*  .‘an-i  prominence  operatine  in  a wine  ran,je 
o'’  sitnat ion.n,  Co  «e  go  fall  circle,  ha.";'-',  as  it  were  to  the  parities  ai.pr.)a'h, 

•Vithout  actually  denying  that  whicii  Fiedler  has  no  persuasively  written  - who  -dares  - 
t-iere  is  -a  need  to  explain  w'ny  somi?  men  ire  mor-"'  freciuently  leader  , re.yardless  of 
'i  cumstairie,  whilst  oth'srs  are  not.  Let  us  then  look,  briefly,  at  t.nis  twin  .ia?3tic:. 
of  ability  and  of  style. 

But  first,  perivip.n  you  will  penoit  iiv»  to  indulge  in  some  intuitive  experiential 
T v'  hology.  and  be  do-?:ii‘itic.  Furry  (l!?73)  says: 

"Leaders  are  all  bright.  Cometimes  they  are  brighter  th.an  others,  sometimes 
they'  are  positively  clever,  sometimes  sh'ewd,  may  be  understanding  a.nd  perceptive, 
possibly  may  bo  possessed  of  inaiglit,  can  see  wood  for  trees,  can  get  to  the 
roots  of  a problem,  can  devise  alternative  s.jlutLons  to  problems,  occasion.nlly 
may'  evoii  be  imaginative,  but  always,  in  one  way  or  another,  are,  mentally  bright" 

c.  Introduction  to  Thought  and  Farsonality,  Ed  .Varr  P 3,  Penguin  Books  1970. 

7 See  A th'!ory  of  leadership  effecti  v-eness,  Fielior  F E,  McGraw-Hill  1?o7.  For  a 

b.*io!>r  presentation  see  Leadership  - a n'w  model,  Fieiler  F S,  discovery'  1965*  Or 

3'^e  ne  .-ontin -eiicy  uio  lel  ani  the  effect  i-f  Leadership  training  and  experience,  Fieiler 

F .1,  in  Pru.'.eedlngs  of  HAT.)  leadership  o.-i.nposiur.-,,  Aae  1 »7\ 
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In  190'U  13  no  h.we  seen,  iniellLi’noe  '".id  wjt  fjjc  bit  th  loi  i;j.> 

'if  thti  anbjeot,  far  t ; vt  it  jaickly  b'jcaai,  ^-rev  .r  2y  t'oo  I faO’a  it,  / i.j  "oiiible 

for  Burt  to  say  without  feir  of  contradiction: 

"fh^  de^7l•ee  of  in  tol  licence  witn  wiled  an,.  ;;.ii.  io  enio70j  ij  cno  ,>f  \'-o  ~.j  t 
idpo.-t  int  f ictnr.i  letonnining  hi.  ’Onoral  'i  i.‘/  ail  throi.jii  >.i t " 

-ti  l for  ’’.jriija: 

'''Thia  aeano  that,  .’.liriy  ;jo,l  preii;ti..n-  ’■  .b.lity  in  o-ii.-ati  n,  ii.i.it:-  r 
everyday  life,  can  bo  achieved  by  'g'  teats  alone,  and  that  soae.vhat  -lore  , -round 
can  be  covered  by  testa  of  the  main  group  factors"  (l0). 

And  it  is  perhaps  notwithout  interest  that,  after  the  Central  Committee  of  the 
Comraunist  Party  of  the  Coviet  Onion  hid  addressed  itself  in  193S  to  the  merits  aul, 
more  specifically,  the  demerits  o'"  ps^'chometrics,  psychological  tests  were  banned 
de  jure  in  the  USSR,  but  they  are  now  used  le  facto  in  several  contexts  including  space 
research  whilst  among  the  tests  are  elements  from  the  best  seller  of  all  time,  .-..aven's 
tiatrlces.  (ll) 

In  short,  we  may  say  that  intelligence,  whatever  it  is,  does,  broadly  speaking, 
predict  potential  occupational  status. 

But  there  is  also  more  recent,  and  er^ually  lompelling,  evidence  that  abo'ie 
certain  cut-off3  and  within  certain  bands  of  Intelligence,  Ij,  does  not  predict. 

Guilford  noted: 

"^^ith  about  fifty  intellectual  factors  already  known,  we  may  say  that  there 
are  at  least  fifty  ways  of  being  intelligent"  (12).  (And  many  more  of  being 
stupid). 

Hot  surprisingly  then,  though  we  may  Bay  intelligence  predicts,  with  as  much 
verisinlllitude,  it  may  be  said  that,  in  some  contexts,  it  does  not.  In  particular, 
intelligence  neither  predicts  excellence,  nor,  as  we  know  from  many  military  studies, 
is  it  of  much  help  in  the  selection  of  potential  leaders, 

0.  And  not  least,  to  be  fair  to  Spearman,  because  of  his  picneering  work  with  t-^trad 
differences  and  factors. 

9.  The  evidence  for  the  concept  of  intelligence,  Burt  C,  British  Journal  of  -Educational 
Psychology,  Vol  25,  1955. 

10.  The  .structure  of  luman  ibllities,  /emon  ? i, 

11.  See,  To  test  or  not  to  test:  tren  Is  iii  tne  >-/ie*  news,  Joseph  3ro  , i ' ^f 

ttie  History  of  the  Behavioural  Set 'n.es,  Vol  J > , 1172. 

IB.  Three  faces  ef  Lii telle' t igaui. 
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The  paradox  is  resolved  like  this.  As  traditionally  measured,  intelligence  in 
all  its  many  and  various  forms  was,  despite  that  variegation,  always  measured  as  a 
'convergent*  ability.  The  tests  had  a correct  solution  which  the  testee  had  to  find 
thus  a premium  was  placed  on  conventional  thinking  ani  on  conformity.  In  the  fifties, 
however,  there  developed  (and  grew  apace  'till  it  became  a runaway  bandwaggon)  a 
whole  new  field  of  intelligence  testing  concerned  with  abilities  of  a different  kind, 
■/ariously  named,  'creativity',  'originality ' , 'divergent  ability',  'lateral  thinking' 
and  'co0iitive  complexity'  these  new  found  abilities  or  styles  of  mental  process  were 
thought  to  explain  the  variance  not  accounted  for  ly  the  conventional  (convergent;  tests, 
certainly  there  is  a growing  body  of  evidence  which  suggests  this  is  bo.  (13) 

This  literature  (or  movement)  had  an  importeint  corollary  in  ttiat  it  became 
inextricably  mixed  with  parallel  deveiopmonts  in  studies  of  flexibility  (versus 
rigidity),  concreteness,  ambipyity^  intolerance,  dogmatism,  conformity  ind  eiuthorit- 
arianisa  - all  it  might  reasonably/ thought  primarily  matters  of  personality  rather 
than  of  ability.  Jo  in  this  growing  and  indeterminate  field,  part  ability,  pirt 
personality,  we  find  mixed  together  tests  of  risk  taking,  tents  of  category  width, 
tests  of  remote  associations,  uses  of  an  object,  measures  of  fascist  tendencies,  :ioe's 
'■‘ninent  scientists,  and  the  contrary  imaginations  of  Liam  Hudson's  schoolboys  whose 
highest  scorer,  you  recall,  wanted  to  stick  nails  through  the  sides  of  nis  barrel,  p(.t 
iunty  inside,  and  roll  her  down  a hillj  Herein  (not  in  the  barrel)  lies  the  key  to  the 
, roblem. 

Locically,  I suggest,  ability  ai’.  i -..er'onality  ;ire  not  and  cannot  be,  indenyndejil. 

.'•le  evil  intent  to  put  aunty  into  a position  of  discomfort  must  n^eds  be  matched  iy  th<- 
inventiveness  necessar;/  to  conceive  the  vehicle  in  which  to  do  it.  similarly,  a 
.■ersonality  properly  described  ac,  'm'.n;i>nary , calculating,  sharp  x'.d  shrewd'  c.'auK.t 
<i.st  without  the  ability  to  make  evaluations,  manipulate  monetary'  data,  comprehend 
md  assent  rtlative  advantage.  The  i.tv  le  Is  necessarily  dependent  on  the  capai>il  j . 

Ir  t}  (,  otifti-  <U>-eetion,  since,  in  general,  co^dltive  skil]  or  alalitj  Joes  not  peis  1 . t 
I'rij;  in  a vacuum  of  disuse,  tiiC  ability  to  perform  in  a certain  v<ay  deiendt  on  tije 
. .cthe  i:,,lif  i by  an  iiabitual  pattern  of  that  behaviour  whicli  j f-  generally  calif  u ^ 

' . . giosi  ti' n ' or  Miy  be  railed  'personality'. 

. Names  to  look  for  in  this  literature  aret  detzele  and  Jackson,  Anne  Roe,  Mckinii'n, 

• M'l  Hudsoi,  ind  on  cctyiitlve  coir])lexi1.,  , see  Changes  in  students*  peifoi'mance  on 
i 'rsonality  tests  during  university  courses,  Abercrcanbie  !•'  I-  J , et  alia  JLRC  repori  , 
a-  ■ v 1 hy  Ctringer  ?. 
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'.Vhat  I want  to  aat^est  is  this.  When  Fiedler  talks  of  leadership  style  he  must 
necessarily  ijnply  the  ability  to  behave  in  that  way.  Purtheimore  the  ability  wliich 
his  ASO  score  measures  (or  at  least  taps)  is  a facet  of  cognitive  complexity.  Those 
leaders  who  Fiedler  describes  as  task  orientated,  because  in  theii  perceptions  of 
people  they  see  wider  differences  (low  LPC  scores)  ttian  do  those  who  are  relaticuishlp 
orientated  (high  LPC  scores),  must  have  the  ability  to  differentiate  implied  in  the 
concept  of  cognitive  complexity  in  order  to  be  able  to  adopt  that  style.  They  make 
those  discriminations  because  they  are  able  to  make  them  and  have  a choice  whether 
they  do  so  or  not.  Those  without  this  perceptual  acumen  cannot  behave  in  tJiat  way, 

Jioreover,  I think  it  may  be  airgued  that  a whole  host  of  leader  ajid  manager 
functions  depend  on  this  critical  ability  to  discriminate.  Shrewd  delegation  of 
responsibility,  and  the  perceptive  and  differential  handling  of  men  are  but  -wo 
examples  dependent  on  this  ability  to  make  accurate  discriminations.  Therefore  1 t.iink 
it  possible  that  in  this  new  arena  of  intelligence  testing  there  is  the  potential  to 
explain  some  elements  of  leadership  by  the  use  of  an  objeotivw  measure  of  this 
cognitive  complexity.  Let  me  describe  how  I tried  to  go  about  it. 

Li Bcri ml nation  Test  Construe blon 

Officers  newly,  and  for  the  first  time,  posted  into  poaltlonii  of  reeponsibility 
for  training  other,  more  junior,  officers  attend  a course  some  of  which  is  concerned 
with  problems  of  appraisal.  Apart  from  lectures,  demcn strati on s , eond  discussians 
they  are  required  as  a training  exercise  in  appraisal  to  write  OERs  on  LPCs  and  MPCs. 
The  data  derived  from  these  exercises  are  additionally  used  in  a continuing  programme 
of  research  into  person  perception  and  the  factors  aiffecting  adequate  performance 
appraisal.  In  particular,  a difference  score  (the  inverse  as  it  were  of  Fiedler's 
ASO  score)  is  calculated  from  this  OER  writing  exercise  ty  simply  taking  the  sum 
(of  the  moduli!)  of  the  differences  between  LPC  and  MPC  rating  on  each  of  the  27  rating 
scales  in  the  OER.  (14)  These  sums  over  27  scales  varied  from  as  low  as  31  to 


14«  For  an  erudite  discussicn  of  difference  scores  see  Proposals  leading  to  analytic 
treatment  of  social  perception  scores,  Cronbach  in  Person  perception  and  inter-personai 
behaviour,  Ed  Tagluri  & Petrullo  1958, 
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aa  high  as  136. 

After  one  week  together  the  course  members  engage  in  some  simple  socio-metrics. 
Inter  alia,  a hypothetical  project  is  described  and  a leader  for  it  chosen. 

Interestingly,  correlations  between  these  differenoe scores  and  peer  selection 
as  leader  of  the  project  have,  with  one  exception,  all  been  positive  and  significant. 
Relating  this  then  to  our  thoueJits  on  cognitive  complexity,  it  seems  at  least 
possible  that  ASO  scores  or  difference  scores  mi^t  be  used  as  a simple  and  objective 
measure  of  leadership  potential. 

If  the  'ability'  6ind  willingness  to  perceive  differences  in  others  is  an 
attribute  possessed  by  those  likely  to  be  selected  as  leaders  in  Infoimal  groups 
of  officers  ly  definition  themselves  experienced,  competent  and  at  least  moderately 
successful  (otherwise  they  would  not  have  been  selected  to  engage  in  officer  training), 
then  here  is  something  - call  it  ability,  style,  personality,  cognitive  structure, 
perceptual  acuity,  what  you  will,  piragmaticadly  it  matters  not  - which  may  be  easily 
eind  objectively  measured  and  as  easily  scored.  But  nere  simple  L.KJ  and  I4PC  and  the 
derived  difference  score  to  be  used  in  the  selection  situation,  there  would  be  two 
obvi ous  drawbacks  t 

Once  the  word  was  'out',  as  very  soon  it  would  be,  scores  could  and  doubtless 

would  be  fadced.  All  LPCs  would  rate;  1,  1,  1,  1,  whilst  liPCs  would 

be  halo  9 men. 

Even  if  this  should  not  happen,  the  test  would  lack  standardisation.  % LPC 
is  not  yours.  Neither  do  we  share  an  U?C,  It  could  legitimately  bo  argued 
that  true  differences  between  L and  MFCs  must  necessarily  vary. 

It  seems  from  this  that  a requirement  be  stipulated  as  follows.  The  abilitj' 
to  differentiate  between  people  may  be  measured  bys 

A standardised  test  which  is; 

Ueaningful  to  the  target  population  and  which; 

Exercises  the  'ability'  to  perceive  true  differences  and  discriminate  between 
people  in  the  way  t>iat  LPC/lJIPC  ratings  'test'  that  ability.  Furthermore  it 
should; 

Include  'cheat'  detectors,  ie  scales  for  which  no  evidence  is  available  and 
therefore  IK  is  the  correct  assessment,  and  sceLles  in  which  the  relative 
positions  of  the  persons  being  judged  are  reversed  as  compared  with  their 
positions  in  the  majority  of  cases.  And  finally; 

The  test  oust  have  a known  'solution'  derived  frojn  research  findings  of  the 
relationships  between  behaviour  and  attitudes  described  in  the  test  with 
subsequent  performance  either  in  training  or  in  an  employment  environment. 
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And  It  raist  sati".'/  noraal  t°3t  crit^rLa  lik--  .Jtability  an  1 re  • i t .c  i r>L  1 1 ty 

and  30  on, 

3inoe,  in  the  Ion;;  nm,  uue  of  t.iis  particular  teat  was  intended  for  a 
particalxr  aelection  :3ntre,  too  intention  win  '’o.-nei  to  oonstract  a nirrit.v' 
type  iescription  of  two  oaadi dates  at  that  cent.rj,  ie  person:  in  exictly  th'='  rti- 
position  as  tae  target  population,  in  wliich  Lnfora.tion  couil  bo  used  by  t le 
perspicacioas  subject  to  User iiainate  between  toe®.  The  inforn  ition  in  th  ? tost 
was  to  be  obtained  by  reference  to  research  finiinjs  so  that  there  would  be  a D3 
answer. 

Such  research  evidence  exists  in  American  literature  in  abuniianoe.  You  will 
recall  the  famous  'luestion,  "Did  you  ever  build  a model  aircraft  that  flew?",  which 
is  reputed  to  have  been  almost  as  good  a predictor  of  success  in  flight  training 
during  V/orld  Yar  2 as  the  entire  air  force  battery.  But  how  far  one  i entitled  to 
take  such  evidence  across  the  cultural  divide  between  one  mglish  speaJcing  nation 
and  another,  is  a matter  for  conjecture  and  possibly  doubt  - thtough  "ecent  cross- 
cultural  studies  have  suggested  considerable  similarities  in  the  value  systems  of 
ciigland  and  America  (l5)*  Necessity  being  the  mother  of  invention,  it  will  be 
shortly  shown  that  compromise  was  made,  iiaglish  evidence  actual  derived  from  the 
selection  situation  in  which  the  test  might  be  usei  was  relied  on  to  the  extent  of 
its  availability  whilst  finlings  from  other  environments  proviiei  supplementary 
material. 

Smith  and  Jones 

Perhaps  before  going  on  to  iescribe  the  substance  of  the  test,  I may  briefly 
remark  on  the  apparently  simple  matter  of  putting  a name  on  it.  "l^hat's  In  a name? 
That  which  we  csill  a rose  by  any  other  name  would  smell  as  sweet",  is  poor  counsel 
to  the  test  designer  engaged  in  finding  a title  for  his  instrument.  He  ma^-  wis'n 
positively  to  mislead  his  subjects  - as  is  so  frepiently  done  when  personality 
measures  are  thinly  disguised  as  perfora;ance  nr  ability  checks  - or  (and  in  v view 
more  properly)  he  may  merely  wisn  to  avoid  gratuitous  antagonism  at  th  ; coiEioncem^nt 
of  his  work  and,  by  the  selection  of  a neutral  title,  happily  and  honefuU.  voil  1 
wish  to  minimise  bias,  prejudice,  anl^or  emotional  set.  In  ajiy  case,  som"  -are  is 
necessary, 

■b,  I->u  p,;r.ivia'.  ;iluJ  nysiem  nejn ’ers  - a > ••  . .t,  hglenl  > \ ; i 

beminar  on  c:-o33-cultural  jtuiie.i,  Jrun  leis  An  1 Accelerated  mun'^r'  M 

in  twelve  countriej,  Bas : X'  et  ilia  in  In  i ut  rial  .uXitions,  <oi  1.’,  . ■ i-. 
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In  if  not  elsewhere,  the  names  'Smith'  and  'Jones',  especially  in 

simple  juxtaposition  Just  like  that,  are  almost  synonymous  with,  'Mr  Average', 

'the  mail  in  t'ne  street',  or  as  Hnglish  law  heis  it,  'the  man  on  the  Claphara  omnibus'. 

The  names  are  emotionally  neutral  and  flat:  certainly  they  are  eiual  - rather  like 
the  advocates',  Roe  and  Doe.  By  contrast,  to  have  chosen,  'Bannister  Uainwaring- 
Jii^/the'  for  appraisal  alongside  simple  'Jack  Jones'  might  well  invoke  the  intrusion 
of  social  class  overtones  and  invite  bias  - error  variance  without  ccaitrol. 

So  simpJy,  ' imi  th  and  Jones  - Judgement  Sxercise'  it  is  and  honestly  described.  kU, 
Instructions  are  given  as  follows: 

"Smith  and  Jones  are  two  officer  candidates.  Both  have  been  tested,  interviewed, 
aiid  extensively  observed  in  small-group  and  individual  exercises.  Your  jot 
is  to  make  an  assessment  of  each  using  the  scales  provided  and  finally  to 
decide  whether  either  or  both  should  be  accepted  for  officei-  training " 

Instruction  on  the  use  of  rating  scales  is  given  in  a manner  not  unlike  tliat 
•iven  in  a number  of  other  stuiies  in  this  field  in  which  two  p»?rsons  are  rated  ur.ir..- 
ne  fona.  The  rating  scales  to  be  used  are  given  and  include  such  dimensions  as: 
mtlvation,  academic  potential,  sociability,  aiid  so  on. 

The  narrative  then  describes  Smith  and  Jones  simply  an  i sequentially  as  the 
' n f onaat  i on  would  be  obtained  an  i recorded  in  the  selection  procedure  with  wliich  laj' 
experimertal  officers  are  fa:ailiar  - having  been  througli  it  - and  with  which  the 
, 'tential  us.n-t,  of  th*  test  would  be  becoming  familiar  because  the  test  is  inter*  led 
b*’  a , . it.  The  narrative  has  five  major  parts  of  w:iif*  1 wall  describe  jur'' 

C:'.ree: 

Infi  tio:i  r.ic  lined  at  flr  .t  interview 

R.-Siitii  o'  yiap^r  'in  1 p?-u-i  1 , -.''st  lOanai  re. 

Ob  ,f.  V itl  oi.  ,1  ti.e  leaderl»'ss  grou;  .iisciision  exercise. 


1o.  For  intt.T  'st in  • lis'ussions  of  the  very  seriixis  issue  of  privacy  -and  restai  cl* 

. .-e  J3RC  lie.vsl-'f  ter  If  Mir  7J,  the  Younger  he-nj-t  Jmnu  **at*il  jn  il.Mo  and 

,■  nerally  on  the  ethics  of  selection  see  The  i syclioloir,-  of  Jelection,  ..'ilson  J ii  in 
.iinigeraent  To'lay,  Jar  1h7  i. 
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Interview  Jata 


?roti  th»  firjt  intervi<=-w  it  id  leaniel  tint; 

A -p  1 19  ' Jmit'i  i ? the  fourth  u i iHi;i."-dt  aoon  o.'  ..ol  1 ■ ; 

el  trici  vi 

"At  iiif;  loinl  - -lenui v-^  ich  ni  ib*  lin ‘ti  (examination  p id.v- ; 
in  at  drTP  

"At  (age)  he  passed  (higher  level  examinations ) in  

"He  played  (various  games  at  team  levels)  

"'Vas  a school  prefect  

"And  an  active  member  of  a loo  il  youth  club  " 

And  then  Jones’  background  is  similarly  described, 

V/e  know  from  the  McGill  studies  (17)  of  the  weight  attached  by  PSOs  to 
achievement  data,  whilst,  more  recently  Martin  (I8)  at  CAP  PARU  has  confirmed  this 
interest: 

"Officer  applicants  tended  to  be  assessed  on  the  basis  of  academic  achievement 
about  which  interviewers,  it  seems,  could  agree" 

"Academic  achievement  correlates  higiily  with  luilitary  potential  

the  PGO's  prediction  of  i canlidate's  potenti  il  for  remaining  in  the  Jervlce 
and  progressing  in  ranic  is  based  largely  on  the  cjindldates  aeaJeuic  ichievement 
at  the  time  of  the  interview". 

Moreover,  the  evidence  reviewed  b>-  Asli.=  r ( 19,1  of  the  better-than-most  predictive 
value  of  historical  and  verifiable  biographical  data  suggests  this  strate.n  both 
prudent  ani  wise. 


17.  decision  making  in  the  en;loj''nent  inter/i  7,  lepster  E C,  Iniuslrial  rexitions 
centt>>,  McGill  'Jniversity  1964. 

18.  r.oliabillty  ani  validity  of  JAP  ^election  interview,  procedures,  ...artr. 

P.eoort  ,'J-d,  JPPAK'J,  May  m>\ 

19.  -'..a  b.  ) "rarhii' li  it-T:  u.  • , nt  ;■  V‘  ; A M.-t  J .1,  . rsoni.’-i  , . t ' 
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it"  T-  <*r*  i:c‘ORiPd  r^ftsonnblp  -ir.ii  f^i  t that  tht-*  ii. 

1 .I'l.i  . nn.'f<"  should  provide  a sininle  res.ino  of  r.-unily  b(iCki7routi<i , si'holsstic 
-i.ihi  >-vcm-i  iPfiiiho  rship'  of  -irih  t-’lHtus  in  oliihP,  ■^lOiotifa  arid  para  nj  li  ^ 

■Tii^a#  ft  i.  alsc  tnio  of  cout’S'^  ti  at  trr'  statistii  al  ai-socaation  of  tt;os'' 

1-  .aio  j |-i  officor  ti-aininr  in  ::i\  ovm  o;  i'arii  -ati  on  is  Vtiovu:.  ftiu-=  : 

I ' :,-i'a‘ii  n'  to  thiP  part  of  tho  p-ohltni. 

Tao  a rra  1 i VI'  rofS  on  to  Imild  a ■ iot'."’-  o'  tt:o  rr»'f'  ?'i  n<'"-.  t;  i ■•  ■■_  v.  -aii  i v.-. 
’ 3;,.  ' t a u,  i '1M-:  . 


l i-  i . ■ ■■'■  y n non 

lila-i!  


i'  ij  I itii.nrostc 


■i  i tl  .1  i,c 


i'.-’  t ■ 


laior  t’  t ivi'vi  of  inn  ort'i! 


■■  1 . . •■•1  

"ilt'.ouaa  hp  had  dori'  n*'’.  tl  or  h<-  tl  . h-  wiauld  like  

" ;p  i-v jiv<- ■■  .P'i  ■■.  li't  i.'tp  for '•n'l  so  on. 

.;i'si  vi'oK  mi’n-'  • iikf*s  ai'ri  iTttor»*.ts  tost  li-ui  provioualy  p.'tiihl  i fhi*a 

•-■.'f;  aril  '■|•.|’I•‘''lati  on:  hPtwp- ; r.i.i'  .y;>-i-paion  ol  thoso  i rr-fpr' ri'-P'.'.  1-"'  nrn'  i -at' 

, ,i  - I..  ;-,i'a  ►■-in  in  traininy.  .'I  ’ ti  if  - is  a.-a-n  ■<  ' -r. ii-sot'  ■'c  l u I ' on . 

, T.|  OT  I'la  i T ■ 

I '1  < ■ fi.i--  i.;a. '►■  i 1 f i Ol  •■twi-fi  r-i-ca.,  in  Irainiin.  u a ’•I.  - I'Ji  '.ui-r;:  f.i  '■  ' 

1 I ■■  t ionnaifi  iircifiy  i..‘  ► -riiai  t,  iru'ioir  a '•■.at.  js  v- 

' |C  >!•  i 1 ■ xii'i  .’ort'o  narr.ativ*'  thiTpf.-'"  inclnipo  'x  in.tlii  t.n''  ■ i :■•  .*■•■  ‘atioi 

■ii.i  o'"  tr--'r..  .|u..f,tions  an  1 tiii'  '■•r.r,*'-'  ‘ suiijosprlly  g'ivi-yr.  i-ij-  r i 

*■^.11-^1  ■ ■ ’’iroon  r'roblPiT'  -lolv.iti  .ii'-i'ii'’--i  on  r 

Thi-  n ic."xtjvi*  also  i rii  i li  i*‘i'  x .'imniatj  > f trs*  w.’iv  in  wiijoh  'mi'  rn  .hii.i''  t 

i r.o.iii'i  I I '111.  : . ■ a'/'.  I !i,  j :'l  c ■ ' n'l.’.i'’,  ■ .vln,'  :ii.  is  io,  ■ . 


,'.»■)  nr  7.  V'lj  , 


.■'■xr.;  ) j ..  r ■•■.ii'i  ...-na.  of  f’l.-  •natnri  x'l : 


. g''x  ||•■!t  a- 1 1'l.  a.noij-t  of  Lni.-  aic  othe:  : •■rsenaj  ity  maanur' ..  sep  I'a-  Moa.'; 
'.V.ti  7-it  i.oT.  m tho  dPip  tiori  ..f  Airori'W,  P.a:—  .)  h,  Joion-r  ’fv, 

• • iV.pr  1 *• .'  . 
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” nr  ki*  I'r"  ;ur;  '-r.  

"hi;<  .-('nf. li  ;»it. ion!  sprt*  1 ,r.*»'ly  ,■  •••iui'i!  jo  • ■ . tion.i  , , • ' >•' 

hivf.  p-\<'n  ‘■•’.■srn  MVP  hij  k',  i^ion',  *n.,  in:  nu  Ht  i or , o;  u 

krnvi  ui  in  to.T  it  i VP , .>  • ■ p-c;  to  t;;  i’  

If!  i Tiii  j mi'lii'y  ^;iol  .■>  • Ju  ■ ''  y,^}) 

Aii  tiiis  ini'oM.i  ition  ■■  t'ar  i ■ i ■ t:  - iir.-  ■/ y . .'•■.,it  i ; t . ^iv, 

the  'goo'iie'  the  bettP.-  inhip  vemerit  rit  i,  y h 1 1 ,t  ia  net  too  - ijy  tc  joe 

this  and  in  diseussiun  there  is  consi  leratile  arnamont  about  its  weie:ht  , vbj.lst  iiij 
likes  and  dislikes,  attitudes  ani  xotives  are  ti.ose  associated  - as  far  as  ne  know  - 
with  'better'  candidates,  ru-ther,  his  supposed  performance  in  the  siiaII  ^roup 
Uscussion  and  in  the  hantrar  exercises  (wJdch  I have  not  described^  is  intended  to  be 
compatible  with  prominence  in  these  situations  mi  with  the  exercise  of  influence„ 

But  lest  this  make  the  discrimination  task  unduly  easy  the  natter  is  deliberately 
made  ambiguous  by  the  injection  very  early  in  the  narrative  of  Jescri  pt  i jns 
of  physical  and  facial  appearance  known  tc  have  a diametrically  opposite  effect  to 
that  of  the  personality  aind  perfomance  material. 

In  1955  Johnson  (24)  lid  some  interesting  research  in  which  she  described  to 
subjects  two  opposites.  The  'baddy'  was  ruthless,  brutal,  hostile,  boorish,  vulgar 
and  so  on,  wliilst  th.e  'goodie'  was  wam-heartei,  honest,  intelligent,  resvcnsible,  etc, 
ohe  then  asked  her  subjects  to  rate  the  like.inoud  thit  these  two  w.'ul  1 lave  certain 
phi’siognomio  features  an  1 f:und  agreement  between  aer  3's  that  her  goodie  an.  :,ad.iy 
would  indeed  look  significiuitly  iiffprent.  f'  inject  jcme  potential  ac.biguity  into 
Jmith  and  Jones  I therefor--  attributed  Johnson's  baddy  to  ny  goodie  and  vice  v‘>rsa. 

(And  it  is  extraordinary  new  many  of  cy  subjects  are  very  much  c.-nfusel  h-y  this  pui-ely 
physiological  an!  physiognomic  lata).  f 


2t.  See  An  Analysis  of  the  leaderless  group  discussion,  Bass  B M,  Journal  of  apnll-d 
psychology,  ’/ol  3',  cn  partici iiat ion  time  and  intluence, 

22.  On  prcK'edural  sus-gestions , see  dom-  Factors:  in  the  selection  of  1-^ a l-r  rn.- 
members  of  small  groups,  tiirochi  --t  alia  J.xiiTial  of  abnormal  and  socin  • .-g,o:.ology , 
Vol  30,  1939. 

23.  On  integrative  behaviour  :miu  lefviershin  s-^e  The  Behaviour  of  Leailer:  ml  other 
.Toup  members,  Carte:-  C •■■t  ali  1,  ■'  lumal  1'  inic  -nai  an  i social  psyc noiorg.’ , <o' 

M,  dee  dter-iotypea  md  nerieptuaJ  li  < • r:-‘ . 'n  , m unpublished  honour-.-  , ... -r/ 

-tihinfetin  College  I ''.a,  Icrncen  -:-'.t-n.-i  v->ly  quoted  by  decord  in  1-  " .n  r-ert'.on 

!!•  t Int-' r-''er  lonal  ;ie;-,:f/i  -u--,  .1  igiuri  ui  1 :eti-ulio,  1933* 
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Ji.Hcudaion 


•luat  b i:oiiraes,  appi-oximately  150  officf^ra,  have  now  atteir.pteri  this  exercie.;, 

A sccrint*  schedule  permits  minor  discrepancies  from  the  correct  solution  but  penalises 
the  following: 

Smith  ar.d  Jont-s  in  reversed  positions  on  any  scale 

..;t!ier  ->mi  tli  or  Jcrnes  the  wren/-  side  of  tlie  half  way  r.oint  on  any  scale 
r-ither  dmith  or  Jones  more  than  1 rating  in  error  in  either  direction 
•lith-r  .iinitn  or  Jones  given  gradin^ts  on  scale  for  wUch  there  is  insufficioi.* 
in fo i on  to  irntJce  a judgement. 

1 

io'sul  ts  •;>  far  mav  be  summarised  as  follows.  Just  under  ' a of  sul  iects  have 
rotally  mis’-ead  tne  situation  and  nieferred  the  'wrong'  candidato.  The  majority 
refer  the  'r.  nt'  can  b date  but  tl  ■ acou”acy  with  which  this  n(<*  i .■ 

the  ratin',  ■.•t’-ie.s.  There  is  a yositi './•■  -'crrelation  between  tlies(>  dmitii  and 

■ ores  -iii  1 t,.',e  di  lTer“n‘  e 3cn’'es  obtained  .n  t ie  exercise.  I'iiore  ja  a ;u..  iii.'- 

■•relation  liei-wpen  the  omith  and  Jon- s s -ores  and  selection  h;.'  peers  as  leaner  i;." 

• potrietio.al  reject. 

It  is  t'ie-"efore  concludeil  tiiat  there  it.  a sug.’estion,  and  I nut  it  no  higher 
.;a'.  t!.at  at  t.iis  stage,  that  therf  is  a tniit  ability  to  jieroeive  differences  betw.-e*, 
peopl  ' an  i t use  scales  to  record  tnose  perceptions.  This  ability  to  di  fferortirt.. 

•V-'  be  .oi..,.illv  related  to  leadn'shi  p potential.  At  least  they  appear’  to  -.o-exiat  .n  i 

■ 1-  .'tati  ti  : il  ly  r.-irr"!  ated . ?•  r.ts  u'si  -ned  on  the  principles  wrr."h  1 ha'/e  nutlinei 

.V  then  iv'.'/e  n note-.Ti-al  as  objective  rieasui-es  of  this  perceiituai  •ihiiit;,  whicf; 

. h.-’  on-'  fast  t of  the  cof^itive  ccn  lexity  or.e  w.nl  I expect  to  find  in  a leader. 

It  I ■ e.-.rly  i'lys  vet.  These  fi’  it  vM.pinrs  after  a little  utrifr'sian  lin.j  cam  ..I 
u i ill  II  t ■.vu-’'.'v;'.t  ror  than  tie  mort  lenl.alive  of  coiu  lut  j ons , lint  of  t.nii.g  1 i 

ir.--ii:if:  1.1  i training  exercise,  if  n.il.  as  -i  test,  dmith  and  Jor.i  s h.as  c n. d .ierai  .■ 

it.'ntial.  -lubjects  who  have  W(irk»  u wi  t'o  it  h ve  testified  to  its  r.;.al  i si.i.  A c.i;iir...n 
•siia'..  e ..  , " h-id  smith  in  11^  .iiuairon",  ivi  1 "presumably  these  are  not  ti.eu  rea . 

I i.'ie.s".  A .mall  minority  in  evei-y  I'linip  so  fir  nuilTin.’'  -i  totally  different  pudgemej, : 

■nr.  *r;'  ...  of  th'"'"  "olreames,  tte  fact  i.f  .i...:...-:sineiit  unreliability  is  forcibly 
.onstr.t.-;  t.i  1 1 -t:  ; i fne  . . th"  [.mbiercs  of  appraisal  j..;  therein  imruiin- . ■ , 

•■on, ’lit  ;.oiv.‘.  (Ani  it  ic  in;/  exr»- nen.-i  tliat  ti.i..  ;'o'’o,-Tii  ti  or:  thrst  ■!  problem  m.ay  ’ 

a neo.  ; .-a.'-  p •'•■"lond  1 1 j on  for  trainir..--  in  lid.  difficult  matter.,  I'he  nnteriaj. 

■’■ides  an  roi.imon  experience  for  n asonod  discu-islon  01.  i n t*  r- person 

•■.•ception  ■«!  I luiigement  .vr.ilst  itruiy  of  the  lessons  which,  must  he  joarnod  if  apiir.'ii.al 
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is  to  improve  m\y  be  drawi  from  it,  ;iei:allinj  our  ■■  loainjiton  laeotlni;:  . ill-Liiis on 

reporting  a 'JdA?  revinv  of  fV3liati>i  • iplvi ; ■ ii;  1 the  no?!  t>  ie/eiop  iietnols 

for  training  ratera,  [ cooiiien  i 'Snitn  ui  i t.j  you  as  on?  ouoh  m-thol. 

In  oonr.l  laLon  then,  I have  trie. I to  e:<pLain  how  1 tnink  one  of  t-ie  isneota  of 
the  'ne.?'  intelligence  ra;^  perhaps  be  related  to  that  potential  for  leader.^hip  niiicli 
exists  more  in  soaie  men  tha;i  in  otiiers,  I do  not  claim  to  have  developed  yet  a usable 
test  of  this  trait/ability  but  I do  suggfjst  t.aat  the  line  of  enquiry  anil  .ievelopment 
I have  outlined  is  worth  following  in  that  context,  I commend  a dmith  anil  Jones  type 
exercise  to  you  if  indeed  you  should  feel  that  the  problem  of  appraisal  in  its  widest 
sense  is  more  to  be  solved  by  training  tlian  by  the  (oft  repeated)  const ructicsi  of  new 
forms  so  that  we  meor  enhance  the  utilisatioi  of  human  resources  in  conditions  of 
growing  demands.  And  finally,  I would  welcome  your  guidance  and  conaaent  both  foraially 
and  informally;  and  theuik  you  for  your  patience  in  li-stening. 


25»  USAP  ewaluaticwi  systems!  review  and  research  reconBiendations,  Wilkinson  R S, 

Paper  presented  to  fflA  Sept  1971  in  Proceedings  of  the  I3th  Annual  Conference  published 
by  the  US  Marine  Corps,  IVaahington  DC. 


673 


NiW  CulKH'I'l  :■  I IT  tin-  .''iMSlIl  I'illi'Ilt  I>r 

' j 

All  i 1 udi-'S  .111(1  Mnl  i vcs  ' 


Robi 

I t ('. . tliiiiili,  .Ir. 

r.' Ri  1 . 

iroli  Orpan 

i z 

a Lion 

K.  1 RdDl'C 

1 lOM 

All 

i 1 uui 

.iii.-i  n.'t  i vi"  ' 

1 : 1 t I'l  1': 

r<  soiit  i.'sp 

ot’ 

i . 1 1 1 y i!  i 

f 1' 

i u 1 I 

i>rob  1 

1 areas 

I'a  r 

a-,  :'l  i p'.v.  1 

li.il.',.  i St  ■; 

. V.'o  c-ia 

i:.c 

asui'o  I'p 

i n 

ioil:.  .iiui 

lu  l i( 

1 : w 

itii  ) 

■,iua 

t pr.  i'  i s i I’ll . 

V. 1 our  : 

sub  i oi'  I s .1 

f L 

c 11  do  til 

i n 

ps  til.  y 

s.iy  1 

in.  y ’ 

i-.Mii ' ; 

: a.' 

, I'l'  il  i ;■  1 i ! .■ 

1.1  .1.1,  .111' 

.1  vico  V.  r 

sa 

. And  . 

VO 

11  V.'lu  11 

i 1 is 

IH-r 

t .cl 

ly  e 

l.-ar  llial  tlu' 

V O'.  'Si  1 

l.'.llll.l"  ilo 

i v.K  ! Il  i r.  ■ 

t : . 

il  on ' l 

• 

I'r.ini.  1 y 

, til 

is  is  a tlKi.li 

ot  i 1 ..  1 J1.I 

I'l  !■ . It  i 

s 

ihoor.  t i 

oa 

1 , not 

1 ro;;i 

t IK 

S.'IU'.I 

a of 

a prt.'i.sc  I'l; 

I'l  U 1 . 1 ion 

, but  1 r.  'V; 

tho  St. IS.: 

pit 

int  o. 

.1  i.'ay 

o] 

l.cik 

i ny 

.It  pi'i'b  Icnis  . 

Ik'v.o  v.'.iy 

I'f  1 L'l'k.  i 

lip 

at.  |inob 

li 

aro 

IV.O  fi. 

r i‘ii  i 

1 I u I 

i 11 

y i 1 1 <i  i 11,;  piT 

lit  i.,1  sol 

u t j I'lis  1 il.  1 

II 

ot  lu  r.'. . 

1 

lii.'po 

1 pr 

C.'K-I* 

1 Sf'mo  v; 

.'\s  of  look  in 

at  .itti 

t 11(1.  : .111(1 

r.. ' 

it  iv.  ■ 111 

lat 

\.'i  1 1 

V i 1 1.1 

ri’K 

(.■  lu 

! ]''  fu 

1 so  1 ut  i I'l;.-;  1 

■'  .1  1 1 f i . u 

It  pro!. 'or 

q 

t Ii.ii:  .'t  11 

li  r 

Tlu’ 

' Liu 

s i s 

of  t il  i : . p.ipi  1' 

i:  ill  .! 

t bi  SI  pr.  b 

1. 

si  . r. 

1 T- 

o;;i  oil:' 

u'.iys 

I’f  ill  in''. 

i 11/ 

.i1h’'.i i .1.1  t i 1 u.l 

iS  .111.1  1 lo 

t i V.  s — i! 

. 1 

I n I up  .ii' 

I'l 

O.'IU  ' i . 

of  till.  VI. 

One 

L'  \:c 

liavi'  .ip|ii'.’pr 

i at  . i.U'.l 

: alu'ul  .It 

t i 

lU.l.'S  .11 

111 

. t iv. 

1 Ik'H 

’an  b 

oth 

ruMsur.  aii.l  o 

liaiipo  till' 

; , 

/ 


TWO  WAYS  OF  1,00,:  ISO  A’)  ATT  H FUFF. 

It  is  d i 1 I i ru  J l 1 1)  d i : 1 U'  ; . t t i I ud  (,  s i I iiou  t )•(  1 1 1 !!;•  t ont  ru- 
vtTsi;il,  or  s.iy  i n;',  si  ou  I iii  ii;-  tiiai  sciif;  i t i vc  pcopli'  ciiijial  to.  So, 
boar  with  ir.o  wiiilo  I woavo  a faiit.;sv. 

Lot  us  imagine  that  tho  U.S.  program  for  space  exploration  has 
established  contact  with  the  third  planet  of  another  star.  Tin's  planet 
is  populated  by  people  who  are  very  similar  to  us  on  earth,  except  for 
one  chiirac t er is t ic . They  are  purple.  After  several  centuries  of  trade, 
travel,  there  are  scattered  throughout  earth  small  groups  of  these 
purple  people,  known  as  the  I’anUees. 

On  the  whole,  relations  between  earth  people  and  tht  FanFios 
are  neither  better  nor  worse  than  relaticms  hetwetn  diffircnt  groups 
of  earth  people  — som.etimes  so-so,  somi.tim.es  terriliK  . Tht  ri  is. 
however,  one  g,roup  which  in  its  luitC’  for  the  Far.kecs  is  notorious. 
Because  this  group  identifiis  itself  with  a specially  d<siened  handker- 
chief, it  is  called  the  llanl;ce  Tankee  I.capiu  (Fd’I.)  . Mem.hers  of  tiiis 
league  do  the  following  with  great  frequency  (som.otim.es  described  as 
a purple  passion) . 

a.  Use  liighly  insulting  language  when  referring  to  Pankees. 

b.  Write  Pankeo  Go  Home  on  walls. 

c.  Avoid  contact  with  Pankees. 

d.  Burn  Purple  crosses  on  Pnnk.ee  front  lav.ns. 

e.  Beat  ind  i v i tiu.i  ] Pankoius,  thus  sp.n  ning,  (h.U  anciint  foil, 
song  about  the  Purple  Peopli  Beaters. 
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f.  Go  into  a foarful  \\\yy  wlionc’Vor 
a ol  .so J i dar i I y . ('Hk-  saluLo  is  a raised 

middle  I'iin^r  extended.) 

liartli  ['.syela’ 1 r>i’ i ,s  t a , altei'  e.in.'lul  study 
madi'  ])roueU'aee!.K  n t s about  me!..l)ers,  ot  t Ite  lii’l.. 
tltev  pos.s>.;.s  ih’juiI  ive  att  iiudis  low.ird  I’.iiikees, 
exh  i 1)  i t a u'idi.'  raiii'i-  ol  Ant  i-l'anla  e lieli.iv  i or  . 
proneu:’;.  I'li.i  at  '•  you  adoiit  will  make  a j’.re.it  deal 
us  see  wliy . 


I'anl.’ee.s  yive  tlieir  salut 
e leuehod  list,  witli  tlu 

of  the  situation,  liavo 
■Sctme  psycho  1 op, i .s  .say 
Still  others  say  they 
Whicli  one  of  tliese 

Ilf  di  f f eriiua  . l.et 
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The  hislory  of  verbal  attitiuie  ineaBurerunL  certainly  prt'Ver.  (hat 
if  you  ask  the  same  or  .similar  qui-.stions  over  ami  ovi'r  apain,  you  can 
get  highly  con.sislent  responses.  In  fact,  most  of  the  iriori'  conmon 
methods  of  attitude  measurement  ari'  based  on  this  fact  (Kd'vards,  1957). 

It  is,  unfor tun.itely , also  well  known  that  verbal  behavior,  no 
matter  how  consistent,  frcciuently  in  not  consistent  with  other  behaviors. 
Despite  this,  the  area  of  attitude  measurement  is  still  devoted  in  large 
measure  to  the  development  of  verbal  consistencies. 

TWO  WAYS  OF  LOOKING  AT  MOTIVATION 

As  long  as  I have  been  associated  with  trainers  and  educators, 

I have  heard  them  plead  for  more  highly  motivated  students.  What  do 
they  mean  by  this?  What  is  a motivated  student? 

As  you  talk  to  training  personnel,  you  find  that  they  describe 
this  student  in  the  following  general  terras: 

A motivated  student  — 

Stays  awake  and  alert  during  lectures. 

Works  steadily  during  laboratory  or  practical  exercise 

periods . 

Reads  additional  books  on  the  subject. 

Tells  the  instructor  how  interesting  the  subject  is. 

Voluntarily  spends  free  time  in  the  instructional  area. 

In  otlier  words  (primarily  those  of  Magcr  [1968]),  the  student  tends 
to  approach  tlie  suljjcct  matter  ratlier  than  avoid  it. 


(Insi’i't  Kif’.urc  3 Alunit  lliTi') 


Fij'ur.  i tii.ii.uMins  lliir.  s i t ii.i  t i .t,  . Tlu'  sMulL-nt  , s t i nii  I .i  t . d I'v 
cho  sv'hov'l  sitiK,Lii>ii,  ri-spoiiiiri  by  ,i  variiLy  ol  forn.-i  of  approach  b.ii., 
vior.  If  VO  u‘-o  Lho  torm  moL  i vat  i on  as  a .‘iliorLliaii'I  tirr.  to  ilo.ar/!-<- 
all  ihvt.-a  boh..viora,  wo  can  doal  w i t li  Lho  siLuaLimi  in  an  (lt<olivo 
I ash  ion. 

(Insorl  Fip.nro  4 AbonL  Horo) 

On  t ho  otiur  hand,  if  we  do  as  slu  vn  in  I'ip.nre  4,  an.l  oonooivi. 
iDoLivat  ioi!  as  Itoinp  inside  Lite  individual,  wo  have  offooLivoly  fir'.ove 
il  from  cnir  view,  anti  are  1 imi  Lt.tl  , as  in  tlie  case  of  atLilnd.s  and 
projU'iire,  Li<  verlKil  upproaelios . 

TUF  IMFORTAhCF  OF  Ili'.llAVlOR  DliSLiR  I I’Tl  ON 
I a::)  convinced  Lliat  a good  case,  can  lie  made  for  tlio  tliosis  tliat 
Lhe  key  Lo  majttr  udvanccnont  in  almost  any  field  of  aj^plietl  psyclioloy 
is  Lite  dost  ripl  ion  of  btditivior  and  Lhe  cties  LhaL  olictiL  or  affect  it. 

idle  L I'chr.  i tjnos  of  lielitivior  desi-r  i pL  i t'n  for  r.ian-m.icliine  systems, 
pioneerttl  bv  Miiltt  and  oLlurs  (lh(,d),  opetitd  the  dt)or  l.o  rsiji'r 
tiilv.iiLces.  in  Lite  prici  tiitn  with  wliicli  m.ic.povor  plar.nini;  for  Lhoso  sy:;i 
c.in  lu’  ctui.iui' L t d . 

Cor.ip'it  ‘ r i Cl  .1  iiietliods  of  analyainp.  d.iL.i  concifnin;;  Jol)  tluties  ,ind 
l.e.l,:.,  iiioi.eiretl  iiy  tdiri.sLal,  Mor.di  (1404)  .uid  ct'i  leao.ues  li.avo  opeiietl 
ihf  V ly  It'  r..i';er  advance:,  in  i>er'innel  m.inc./.oiiient . 
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Tlio  most  important  d i scovorv  that  uc  in  IhimKUO  made  rtmcor  ii  i np, 
the  dcvcl  opniont  of  offeolivi’  and  efficient  iraininp,  was  that  v.diat  is 
taupdit  (the  lu'havior  rtciuired  of  tiie  student  after  instruction)  is 
generally  more  important  than  liot.'  it  is  taupiit.  Tiiis  discovery  had 
profound  consequences  in  tlie  development  of  progranuned  instruction  and 
the  systems  approach  to  training  and  education,  all  of  v;hich  emphasize 
tlic  careful  description  of  the  behavior  requirt-d  of  the  student  (Smith, 
1971)  . 

The  people  comprising  the  behavior  modifiers,  applying  Skinnerian 
concepts  to  a vjide  range  of  problems  liave  developed  many  new  ways  of 
dealing  with  the  maladjusted,  autistic  children,  school  children,  and 
many  others.  Again,  the  key  has  been  the  careful  description  of 
the  beliavior  involved. 

It  is  because  of  this  history  that  I feel  it  is  important  to 
describe  the  behavior  associated  with  the  concepts  of  attitude, 
prejudice,  and  motivation.  This  behavior  can  tiien  be  the  foundation 
for  new  and  more  effective  ways  of  measurement. 

TOWARD  BllItWlORAL  DESCRIPTION’  IN  THE 
AREA  OF  PREJUDICE 

As  a beginning  for  the  description  of  behavior  related  to 
prejudice,  let  us  turn  to  Allport's  (1954)  analysis.  Allport  identifii’ 
five  general  classes,  or  degrees,  which  prejudiced  behavior  raav  tal;e. 
Tljf.sc  are  Ant  i 1 ocut  ion  , Avoidance,  Discrimination,  ITiysical  Attack, 
and  Extermination.  For  ease'  in  comiiun  ic.at  ion  I will  call  tlie  olijc'ct 
of  prejudici-  tlie  othi'r . 
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Ant  i 1 ocuL  i on  refers  to  t.ilkiiif;  apainsL  the  other.  With  persons 


who  think  the  s.nne  way  he  does,  or  on  anonymous  questionnaires,  or  some- 
times with  strangers,  the  prejudiced  person  freely  expresses  his  antaponisn, 
opinions  and  l)eiiefs. 

Av('idaiice.  A more  intense  form  of  behavior  is  to  avoid  members 
of  the  ot  her  proup.  Kven  at  the  expense  of  inconvenience,  tlie  individual 
removes  Iiimself  freri  situatic>ns  in  t-diich  he  will  encounter  the  o titers . 

He  does  not  directiv  harm  the  o titer  proups,  but  simpiy  withdraws. 

Discr  imiitat  ion.  This  is  more  active  than  avoidance.  Now  thr 
individu.iJ  actively  maltes  distinctions  that  arc  harmful  to  the  otlier . 

He  may  try  to  keep  otlu  rs  from  certain  jobs,  rosidcittial  areas,  liosn  i ta  1 r; , 
scliools,  clturches,  or  various  social  privtlepes. 

Physical  Attack.  Under  coitdltions  of  Itiph  fear  or  anger,  .acts 
of  actual  or  syntbolic  violence  may  Ite  carried  out  against  ot  iters . 

Croitscs  may  be  burned;  others  may  be  beaten  or  forced  to  leave  the 
forbidden  places;  insults  may  be  painted  on  walls;  the  others’  places 
of  worsitip  may  be  hombo  1 . 

Extermination.  Tito  final  and  tragic  level  of  prejudice  is  tlie 
killing,  lynching,  or  extermi  nat  Lon  of  tite  other . 

Allport  believes  tint  thc.'te  foiTts  of  bedtavior  leptosimt  a i ouga 
scale  — that  activitv  on  oiii-  level  mai’cs  it  easier  for  preitulicod 
behavior  t(t  move  to  n itigber  Jt'vc'l  . litis  is  one  leasoit  why  it  is 
import  ant  to  noac.ure  tiu  lnhaviors  itt'.'olved  at  1 be  vat  ions  levels. 

Wliat  cait  vi'  til'  to  r,c  asiire  I ite  various  ht'liaviors  ittvtilved  in 

pre  jild  i CO? 
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For  .uit  i ] uciit:  ion , we  cnn  take  on  iip-to-d;Uc  i inxiit  ory  of  oiunion;; 


and  beliefs  c (.nicer  ninp,  ! he  ('t  her  . The.se  can  be  obtained  bv  i liter  v i e\.’ i ne, 
botll  tile  individual  and  tlie  ot  her . ft  would  try  to  express  these  critical 
com/nerits  as  clearly  a.s  pos.sible,  but  7 would  not  subject  theiii  to  aiiv 
attitude  scaling  technique.  Instead,  I would  ask  respondents  if  thev 
had  heard  any  of  those  conments  expressed  within  a short  time  period. 

In  this  way,  a general  index  of  the  amount  of  anti-locution  going  on 
could  be  obtained.  Periodic  administrations  could  then  alert  commanders 
to  the  rise  or  fall  of  anti-locution  within  their  commands,  and  give  clues 
as  to  the  effectiveness  of  race  relations  programs  or  the  possibllitv 
that  prejudicial  behaviors  are  beginning  to  escalate  to  the  next  highest 
level . 

For  avoidance,  1 would  identify  the  most  common  situations 
in  which  avoidance  might  occur.  I would  then  use  this  inventory  to 
identify  the  general  levels  of  both  avoidance  and  positive  social 
contact.  My  suspicion  would  be  that  the  others  would  be  more  sensitive 
to  avoidance  tlian  the  individual  himself. 

For  discrimination,  it  is  recognized  that  many  of  the  most  overt 
and  obvious  forms  have  been  outlawed.  Yet,  from  all  that  I have  seen 
and  read,  there  are  still  a number  of  more  subtle  discriminations 
remaining.  Again,  I would  develop  an  inventory  of  common  forms  of 
discrimination,  and  I would  rely  more  heavily  upon  information  from 
the  others , since  they  would  recognize  many  forms  that  the  individual 
would  not . 

For  physiical  attack,  again,  1 would  develop  an  inventory’  of  the 
more  subtle  and  hidden  fiirn.s  ('f  propt'rtv  destruct  ion,  insults,  and 
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assault  . Those  incitknts  that  attract  the  attention  of  the  ct'nnander 
or  the  military  police,  and  often  the  press,  are  easily  recorded,  but 
are  often  the  "top  of  the  icehcr;;." 

Extermination  is  clearly  not  a national  policy,  as  it  v.ms  apainst 
the  Jews  in  Nazi  Germany,  or  apainst  the  llutus  in  Tutsi-controlled 
Burundi.  Yet,  tin'  race  riots  and  mass  fights  that  have  occurred  on 
military  vessi'ls  and  iiosts  border  on  atter’pts  to  exterminate  small  p,roups 

iVow,  let  me  sum  up.  I believe  that  prejudiced  b(havior  can  be 
measuiH'd  llirouph  the  inventorv  approach  — not  through  the  utitude  seaii; 
approach  — and  I urge  the  .services  to  tlcvelop  and  apply  such  inventories 
on  a regular  basis. 

There  is  a model  for  v^■hat  1 have  proposed.  Tliat  is  the  llumERO 
effort  to  develop  a list  of  d i se r im i na torv  and  irritating  hch  viors 
which  U.S.  troops  and  commanders  displayed  (often  unknov.'i ugly)  toward 
Katusas.  These  were  Korean  soUlierr;  serving  in  U.S.  units  (McCrary, 

19G9)  . This  work  suggest.s  tlie  pe.';s  ib  i 1 i ty  of  using  Llie  inventories 
as  a moans  of  inforrJng  commanders  and  others  exactly  v.'iiat  their 
responsibilities  arc-  in  the  area  of  control  of  prejudiced  behavior. 

In  fact,  I am  convinced  tiril  the  af>proac!i  to  measurement  that 
r iiave  described  iiolcis  the  key  to  the  eleven]  ('pment  of  practical  method.s 
for  tliC'  elimination  of  prejudice.  But  that  Is  another  paper. 

TOV.'ARn  BERWinRAL  OrSCR  ] P'i  lOM 
OF  .STUnF.MT  MOTlVAilON 

Migp  r (I9hd)  li.is  .sugp.e.sted  an  ai'piu'ach  to  the  behavioral 
definitii'ii  of  attitude'  toward  learnini-.  A studi-nt  has.  a favorable 
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altitudi',  or  i^;  1)ol;  i t ivo  1 y mot  i vaL  od  tov.’ard  liarniu;’,  if  ho  ajjproarhos 
tho  subjooL  or  llu'  loarniiv,;  i Lvial  ion . Convorsi'l  y , Ik-  lias  an  unfavorahJ 
attitude',  or  is  lU'j-.at  ive'ly  motivati-d  toward  loarninj',  if  In-  ivuids  t lie 
subjfCL  or  tho  Joarninj-,  situation. 

For  suitable  ineasuronii'iit  of  tlic  motivation  of  stiuK'nts,  v.’o  noed 
to  develop  an  invi-ntory  of  approach  be-haviors  and  avoidance  behaviors 
common  to  students  in  a particular  course.  To  be  most  useful,  the 
inventory  should  be  specific.  Then  the  inventory  could  be  used  by 
either  instructors  or  students. 

Once  these  approach  and  avoidance  behaviors  have  been  identified, 
then  we  can  change  the  student's  environm.ent  so  that  approach  behaviors 
are  reinforced  and  avoidance  behaviors  are  extinguished.  Mager  pre- 
sents a number  of  ideas  for  accomplishing  this. 

SUMfURY 

To  summarize  my  main  point.  We  can  make  good  progress  toward 
dealing  witli  probli-ms  of  attitudes  aiul  motives  if  we  conceive  of  them 
as  shorthand  descriptors  of  appropriate  beliavior.  We  will  make  little 
progress  if  we  conceptually  bury  tlum  inside  the  individual,  therefore 
making  them  inaccessible  exci-pt  tiirough  the  medium  of  words. 
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ADMINISTRATION  OK  MULT  I PLE-CHOICK  TESTS 
to 

NON-READERS  VIA  TAPE  RECORDER:  Case  Studies 

Ji'sepli  I..  Boyd,  Jr.,  Ed.D. 

1. duc  at  Iona  1 Testing  Service 

iiie  Autc'mohile  Mechanic  Certification  Tests  are  administered  by 
Educational  Testing  Service  for  the  National  Institute  for  Automotive 
Service  Excellence.  Tests  are  given  twice  a year  at  some  200  test 
centers  in  all  fifty  states.  The  Program  offers  voluntary  certification, 
based  (in  a combination  ot  work  experience  and  test  performance,  in 
various  service  areas,  i.e.,  "Engine  Repair,"  "Brakes,"  "Heating  and  Air 
Cond i t icm  i ng , " and  as  "General  Automobile  Mechanic"  for  those  passing 
tile  tests  in  all  areas. 

However,  my  objective  today  is  not  to  brag  about  the  program,  but 
to  discuss  a problem  that  faces  us  and  to  tell  you  some  exploratorv 
steps  we  are  taking  to  solve  it.  The  problem  is  literacy — the  ability 
to  read  tlie  test  questions.  Our  initial  approach  was  that  no  individual 
could  be  a competent  mecfianic  if  he  was  unable  to  read  the  automobile 
manul ac turers  service  manuals.  One  principle  of  test  development  we 
adopted  was  tliat  the  reading  and  vocabulary  of  the  tests  should  he  no 
more  difficult  that  the  "well-thumbed"  sections  of  the  service  manuals. 
Not  so.  Wo  soon  discovered  there  are  two  groups  of  mechanics,  manv 
highly  esteemed  by  their  employers,  who  could  not  be  adequately  tested 
by  conventional  paper-and-penc i 1 means.  The  first  group  consists  of 
English-speaking  mechanics  who  cannot  read.  Some  are  marginally 
literate,  but  they  are  placed  at  a great  disadvantage  when  attempting 
to  read  a test.  The  second  group  consists  of  mechanics  whose  native 
language  is  not  English.  They  may  be  literate  in  their  own  language 
but  not  in  English. 

The  traditional  method  of  accommodating  this  type  of  test-takers 
is  to  allow  a translator  or  reader.  We  rejected  this  approach  because 
we  were  concerned  with  test  security  problems  that  might  arise,  and 
because  a satisfactory  translator  or  reader  would  have  to  have  a 


knowledge  of  the  automotive  teciinioal  vocabulary  and  could  inject  some 
of  his  knowledge,  unintentionally  perhaps,  into  the  testing  process. 

In  some  situations,  it  is  feasible  to  translate  the  tests  into  the 
foreign  languages.  For  example,  tiie  Flectronlcs  Inventory  Examination 
of  the  Guided  Missile  School  at  Redstone  Arsenal  was  translated  into 
German,  Greek,  Chinese,  and  Turkish.  This  was  acceptable  since  tnese 
translated  tests  were  to  be  used  at  onlv  one  location  and  with  sizeable 
groups.  On  the  otiior  hand,  for  the  automobile  mechanic  tests,  the  calls 
for  assistance  came  from  everv  part  of  the  country,  and  for  an 
unpredictable  mixture  of  language — Greek,  Spanish,  French,  Italian,  and 
would  you  believe,  Finnislil 

However,  the  greater  in>ed  seemed  ti'>  be  to  help  the  non-  or  poor 
reader.  The  solution  we  arrived  at  was  to  record  the  text  of  the  test 
so  that  each  mechanic  could  listen  to  the  words  while  following  alone 
in  his  test  book.  Each  candidate  would  have  a cassette  player  and  a 
light  weight  headset.  He  could  reverse  the  tape  to  liear  the  question 
read  as  many  times  as  he  needed.  Most  of  these  tests  allow  three  hours 
of  testing  time  but  the  test  text  fitted  easily  on  a sixty  minute 
cassette.  Tlic  one  test  which  was  timed  for  four  hours  required  a second 
tape.  The  liandout  shows  five  typical  test  questions.  This  is  what  a 
mechanic  would  hear.  (PLAY  TAPE  OF  FIVE  TEST  ITEMS.) 

This  system  was  tried  out  at  tiie  regularly  schedule  tost 
,idm in  is t rat  ion  on  April  28  and  May  5 of  this  y-ar.  It  was  simply  a try- 
out of  the  feasibility  of  the  system--ccrtainly  not  a controlled 
experiment.  Four  mechanics  whose  employers  stated  they  could  not  read 
well  enough  to  take  the  test  participated.  In  addition,  we  had  two 
other  mechanics  whose  native  language  is  not  English  who  volunteered  to 
take  the  tests.  These  men  felt  tlieir  grasp  of  spoken  English  was  good 
enough  to  trv  the  tests.  Let  us  take  a look  at  the  results,  case  by 
case  . 

■1 . D.  of  Canton,  Ohio  took  the  tests  in  the  fall  of  1972  , and  failed 
all  tests.  His  percentage  scores  were  at,  or  barely  above,  the  chance 
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level:  J4  , 38  , 35  , 24  , and  26 — for  4-clioice  multiple  choice  items. 
Although  he  did  not  pass  any  of  the  tests  in  the  spring  administration, 
his  scores  were  substantially  improved.  There  were  seven  scores.  His 
percentages  were:  61,  63,  33  (corresponding  to  his  earlier  35)  53,  60, 

48,  and  36.  The  test  administrator  was  enthusiastic.  He  renorted 
that  .1.  D,  worked  his  way  through  the  tests  fairly  rapidly,  though  he 
did  repeat  a number  of  items.  He  also  observed  that  the  candidate  was 
apparently  following  the  text  in  his  testbook  with  his  finger  as  he 
heard  it  read  on  tape. 

A.K.  was  another  mechanic  wlio  liad  previously  taken  the  tests.  In  the 
Fall  of  1972,  he  scored  34  percent  on  "Front  End  and  Brakes"  and  24 
percent  on  "Electrical  Systems."  In  the  spring  series,  using  the  tape, 
he  scored  60  percent  on  "Front  End,"  48  percent  on  "Brakes,"  and  only 
34  percent  on  "Electrical  Systems" — a substantial  improvement  in  two 
out  of  three  areas.  One  might  conclude  that  A.K.  really  does  not  know 
much  about  "Electrical  Systems."  A.  K.'s  native  language  is  Burmese. 

The  four  other  meclianics  who  took  the  tests  with  tlie  aid  of 
cassette  tapes  liad  not  taken  the  tests  in  the  Fall  1972  administration. 
Two  of  these  candidates  attained  high  enough  scores  to  be  certified  in 
a service  area:  J.  L.  as  an  "Automatic  Transmission  (Light  Repair) 

Mechanic"  and  R.  B.  as  a "Front  End  Mechanic."  Three  of  this  group  of 
four  men  scored  50  percent  or  more  on  each  test  they  took — two  times 
the  chance  score.  My  conclusion  was  that  (1)  the  candidates  did  know 

A 

something  about  automobile  repair  work,  and  (2)  the  candidates  were 
able  to  comprehend  the  tests  reasonably  well  through  tiie  process  of 
reading,  listening,  or  a combination  of  both. 

The  final  case  was  G.  S.  who  requested  to  be  tested  in  Italian. 
However,  he  agreed  to  take  the  test  with  the  English  language  tape.  The 
test  administrator  reported  the  G.  S.  found  it  "difficult  to  concentrate 
on  questions  being  read  wliile  at  the  same  time  trying  to  follow  wording 
in  his  test  booklet  and/or  operating  the  tape  recorder — the  use  of  the 
tape  recorder  became  so  cumbersome  that  after  the  18th  question,  we 
eliminated  the  recorder  and  road  tlie  remainder  of  the  test  to  him." 
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Five  of  the  six  men  who  tiiok  the  tests  in  this  wav  regarded  the 
tape  as  a definite  iielp  in  allowing  them  tii  demonstrate  their  competence 
as  automobile  mechanics.  The  Test  Administrators  concurred  with  the 
candidates'  reactions. 

We  propose  to  use  the  tane-recc'i'ded  test  procedure  again  this  fall, 
concentrating  on  mechanics  win'  h.ivi-  previously  taken  the  tests.  Some 
time  in  the  future,  if  the  demami  tleveli'ps,  we  v%'i  I I record  an  oral 
translation  of  the  tests  into  Spanish  lor  those  mechanics  wh<'  are  more 
comfortable  in  that  language. 
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Racial  Differences  in  AFQT,  AQE,  and  WAIS  Scores  of  Air  Force  Basic  Trainees 
Stephen  Knouse,  David  F.  McGre\'y.  and  Ronnie  A.  Thompson 
Personnel  Research  Division,  Air  Force  Human  Resources  laboratory 

Much  recent  emphasis  in  teslini:  researcli  has  focused  upon  unbiased  or  "culture 
free"  assessment.  The  problem  of  U sl  hia*-  aeainsl  minorities  is  especially 
important  to  toda\'s  mililar\-  u’hicli  l ow  mii-t  look  to  true  volunteers  lor  its 
entire  manpower  supph'-  In  tlie  p.i-  t irai  t,  e^  <md  tliose  induced  by  t!u  dral  t 
to  volunteer  produced  a manpower  pi'ol  wit!i  a sut  ticii-ntly  wide  \-ariance  in  aptitudes 
to  provide  adequatelv  lor  personnel  need<~.  Now.  in  the  abseni  i-  ol  dral  t pressure, 
the  .Air  Force  may  !''e  restrii  ted  in  its  manpower  input  and  must  insure  its  selection 
tests  are  tappinc  th<'  full  ranee  of  part  ii.  j)ar]>-  in  minority  groups. 

The  two  primary  si’lection  li’st-  emphwe  l by  the  .Air  Force  arc*  the  .Armed 
Forces  ' lalif ic.ition  T.'st-  i \F\2T'i  ird  th>.  Airman  Qualifying  Examination  (AQE). 
•\dmini‘^tered  by  tlie  recruiter,  tin  \QE  yields  four  composite  scores-- Adminis- 
trative (.\QE-.\),  Plei.  tronic  (Ak'E-IA.  Giiu  ral  (AQE-G),  and  Mechanic.H  (AQE-M). 
These  \ptitude  Indexes  (\Is)  are  U‘'ed  in  the  recrui  tinp  and  initial  assignment  of 
basic  airmen.  * The  .AFQT  is  pnmai 'Iv  usi  d to  determine  mental  category  of 

^\'ote.  On  1 July  197.3  thi>  \rnn'd  s,  j-  -ices  Vi'cational  Aptitude  Batti'ry  (ASVAB) 
replaced  the  AQE  in  the  ,\ir  Force  lestini;  in\entory.  Like  the  .AQE,  the  ASVAB 
provides  the  .Administrative,  Electronic,  General,  and  Mechanical  .Aptitude  Indexes. 
Intercorrelations  betw^een  the  .ASY.AB  and  AQE  ran^e  from  . 59-.  62  for  Administra- 
tive, . 72-.  75  for  Electronic,  . 71-.  7.5  for  Gimcral,  and  . 71-.  74  I'or  Mechanical, 
based  upon  four  pre-seU'cted  samples  of  1000  subjects  each. 
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pre-inductees.  The  test  evaluates  verbal,  numerical,  spatial,  and  mechanical 
abilities,  and  yields  .1  single  score.  Administration  of  the  AFQT  usually  occurs 
at  the  Armed  Forces  Fntrance  and  examination  Station 

Two  recent  studies  illustrate  tlie  direction  of  /\ir  Force  researcli  into 
minority  ^roup  test  performance.  The  problem  of  racial  bias  in  the  AQE  was 
investigated  by  Shore  and  Marion  (1972)  who  employed  the  AQE  to  predict  perform- 
ance on  the  Specialty  Knowledge  Test  (SKT),  a test  of  job  knowledge  used  in 
determining  an  airman's  promotability  to  a higher  grade.  Bias,  defined  as 
underprediction  of  the  SKT  by  .\QE  AIs,  was  not  found  in  the  16  career  areas 
investigated. 

Guinn,  Tupes,  and  .Mley  (1970)  reported  a study  exploring  the  relationship  of 
demographic  variables  on  .Air  Force  ba.sic  trainees  to  several  aptitude  tests. 

They  fourtd  a majority  of  test  scores  were  signif icantly  related  to  race,  educational 
level  and  geographic  area  of  the  trainee.  Racial  differences  were  most  prominent 
in  tests  requiring  specific  prior  knowledge,  such  as  verbal  and  inechanical  tests, 
while  situationally  defined  tests  measuring  non-verbal,  spatial,  and  psychomotor 
abilities  revealed  less  pronoum  ed  racial  effects. 

A further  inve-stigation  exploring  the  relationship  of  the  AQE  and  AFQT  to  a 
test  of  general  mental  ability  appears  necessary  in  light  of  the  evidence  presented 
by  these  two  studies.  The  Weciisler  .Adult  Intelligence  Scale  (WAIS),  an  individually 
administered  test  of  goni’ral  mental  ability,  offers  the  Verbal  (VTQ)  and  Performance 
IQs  (PIQ)  as  well  as  the  Full  Scale  IQ  (FSIQ),  against  which  the  four  AQE  AIs  can  be 
contrasted.  In  the  present  study,  the  \'’AIS  was  administered  to  a group  ol  black 
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Air  Force  recruits  and  a group  of  white  Air  Force  recruits  who  had  taken  the 
AFQT  and  AQE  prior  to  their  entrance  on  activ'e  duty.  Differences  between  the 
WAIS,  AQE  and  AFQT  scores  were  explored  as  well  as  the  predictive  relationship 
utilizing  the  WAIS  IQs  as  criteria  and  the  AFQT  or  AQE  AIs  as  predictors. 

Method 

Subjects:  The  subjects  were  200  male  Air  Force  basic  trainees.  One  hundred 
black  and  one  hundred  white  airmen  at  Lackland  AFB  were  randomly  selected 
over  a 12  month  period  (Feb  1972-  Feb  1973)  during  the  routine  testing  occurring 
on  the  sixth  day  of  basic  training.  The  age  of  the  subjects  ranged  frorr  17  years 
to  23  years  with  a mean  of  19  years.  Education  ranged  from  8 to  16  years  of 
schooling  completed.  Mean  education  le\el  was  12  years. 

I’rocedure:  The  Wechsler  .Adult  Intelligence  Scale  was  administered  to  the  trainees 
by  the  authors.  Because  the  authors  were  members  of  the  military  and  the 
trainees  might  perceive  them  as  rcpresenta tives  of  the  authority  structure  and 
similar  anxiety-producing  stimuli,  military  aspects  of  the  situation  were 
de-emph<isized  as  much  as  possible.  Each  subject  was  asked  to  relax,  to  smoke 
or  drink  a Coke  if  he  wished,  and  informed  he  need  not  call  the  test  administrator 
"Sir.  " Each  subject  was  further  instructed  that  he  was  in  an  experimental  situation 
and  that  his  test  score  would  not  be  entered  into  his  records  or  used  for  job  place- 
ment in  the  .Air  Force.  Each  WAIS  record  form  was  scored  by  the  test  administrator 
and  checked  by  the  other  two  authors. 
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Scores  on  the  Armed  Forces  Qualil  iccUior  Test  and  Airman  Qualifying 
Examination  were  obtained  from  basic  training;  flij^ht  records. 

Results 

Means  and  standard  deviations  of  the  AFQT,  the  four  AQE  Als,  the  three 
WAIS  IQs,  and  the  eleven  VVAIS  subtests  for  the  black  and  white  samples  appear 
in  table  1. 

Table  1 about  here 

t tests  (df-198)  revealed  that  blacks  scored  significantly  lower  than  whites  on  the 
AFQT,  AQE  Als,  and  the  VIQ,  PIQ,  and  FSIQ  of  the  WAIS.  Analysis  of  the  WAIS 
subtests  demonstrated  blacks  scoring  significantly  lower  on  all  subtests  with  the 
exception  of  Digit  Span  and  Digit  Symbol,  wliere  tliere  were  no  significant  difference 
between  blacks  and  whites. 

A correlation  matrix  of  AFQT,  AQE,  and  WAIS  scores  appears  in  table  2 for 
the  black  sample  and  table  3 for  the  white  sample.  All  correlations  for  the  black 

Tables  2 and  3 about  here 

sample  were  positive  and  significant  (p<.  05),  except  PIQ  vs  AQE-A,  VIQ  vs  AQE-M, 
and  FSIQ  vs  AQE-M.  All  correlations  for  the  white  sample  were  positive  and  signi- 
ficant (p  < . 05).  The  intercorrelations  of  the  VIQ,  PIQ,  and  FSIQ  for  the  black 
sample  were  of  a magnitude  equivalent  to  that  of  the  white  sample.  The  intercor- 
relations of  VIQ,  PIQ,  and  FSIQ  with  the  AFQT  and  AQE  composites,  however,  were 
consistently  much  lower  for  black  airmen  than  those  of  the  white  sample. 
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Rej;;ression  equations  were  run  utilizing;  the  WAIS  IQs  as  criterion  scores  and 
the  AFQT  or  AQE  AIs  as  predictors  to  determine  racial  differences  in  test  scores. 
The  three  models  employed  are  listed  in  table  4. 

Table  4 about  here 

Table  5 represents  a summary  of  the  rcg;ression  analyses. 

Table  5 about  here 

Racial  differences  were  found  in  all  reg;ression  problems  except  the  case  of 
AFQT  predicting  VIQ.  In  the  cases  where  PIQ  and  FSIQ  were  criteria  for  the 
.\FQT  and  AQE-A  and  where  PIQ  was  the  criterion  for  AQE-G,  the  type  of 
difference  was  a race  main  effect  (no  slope  difference  between  black  and  white 
groups).  All  other  differences  were  determined  to  be  interactions  between  race 
and  predictor  test  score. 

Discussion 

The  results  of  this  study  indicated  that  only  the  AFQT  in  relation  to  the 
WAIS  Verbal  IQ  demonstrated  no  racial  differences.  The  AFQT  predicting  the 
PIQ  and  FSIQ  demonstrated  a race  main  effect,  while  most  of  the  AQE  AIs 
exhibited  some  type  of  interaction  occurring  between  race  of  trainee  and  his 
AQE  AI.  These  findings  agree  with  Guinn,  Tupes,  and  Alley  who  report  several 
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significant  interactions  between  aptitude  test  and  demographic  variables. 

Significant  differences  between  black  and  white  scores  occurred  on  the 
AFQT,  the  four  AQE  AIs,  and  tlie  three  WAIS  IQs.  Blacks  performed  uniforn.ly 
lower  on  both  tests  of  general  mental  ability  and  selection  tests.  Before  any 
conclusions  on  the  abilities  of  blacks  can  be  made,  however,  a determination  of 
exactly  what  these  test  scores  are  revealing  is  necessary.  An  examination  of  the 
WAIS  subtests  demonstrates  that  blacks  scored  significantly  lower  on  all  subtests, 
except  Digit  Span  and  Digit  Symbol  where  blacks  and  whites  are  statistically 
equivalent.  These  two  areas  are  unique  in  that  they  are  perhaps  the  least  related 
of  all  the  WAIS  subtests  to  the  skills  acquired  through  the  formal  schooling  of  the 
subject.  Digit  Span  requires  the  subject  to  repeat  series  of  numbers  forward  and 
backward,  while  Digit  Symbol  is  a test  of  how  accurately  the  subject  can  match 
numbers  and  unfamiliar  symbols.  Memory,  attention,  and  visual  perception  are 
three  essential  abilities  measured  by  these  two  subtests. 

Guinn,  Tupes,  and  Alley  have  categorized  perceptual  and  psychomotor  skills 
of  this  type  as  situational  abilities,  as  opposed  to  prior  knowledge  abilities, 
which  are  a function  of  an  individual's  educational  background.  They  found,  and 
our  WAIS  data  corroborated,  that  prior  knowledge  items  produce  greater  racial 
differences  in  test  scores  than  situational  items.  Perhaps  there  is  an  over- 
abundance of  prior  knowledge  items  within  the  AQE  subtes' s,  which  are  responsible. 
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in  part,  for  the  differences  between  blacks  and  whites  in  AI  scores.  Future 
revisions  of  selection  tests,  especially  the  AS\/AB  now  coming  into  use,  should 
include  an  investigation  into  the  degree  of  importance  assigned  to  prior  knowledge 
and  situational  variables  in  the  construction  of  the  test.  Such  an  endeavor  may 

discover  a control  for  qualitative  disparities  in  educational  level  of  black  and  ^ 

I 

i white  recruits  which  may  be  masking  the  identification  of  true  ability  in  these 

tests. 

In  addition  to  item  specificity,  a valid  interpretation  of  the  black-white  score 

i 

I differences  involves  a close  examination  of  the  test  interrelationships.  The  correla- 

' tion  matrices  and  the  regression  R s show  a much  closer  relationship  between  the 

selection  tests  and  the  WAIS  for  whites  than  for  blacks.  The  consistently  high 
intercorrelations  of  the  white  sample  indicate  a good  deal  of  common  variance 
shared  between  tests.  The  much  lower  intercorrelations  of  the  black  sample 
evince  that  for  them  the  selection  tests  and  the  WAIS  are  measuring  largely 
divergent  factors.  Further  research  into  the  internal  characteristics  of  these 
tests  is  necessary  to  determine  the  cause  of  this  disparity  between  whites  and 
blacks  in  Lest  commonality. 

Two  possible  areas  of  inquiry  are  the  literacy  factor  and  the  administrative 
format  of  the  AQE  and  WAIS.  The  AQE  is  a paper  and  pencil  test  requiring  the 
i subject  to  read  the  instructions  and  test  item.s.  WAIS  instructions  and  test  items, 

on  the  other  hand,  are  presented  orally  by  the  test  administrator.  Only  the 


Vocabulary  subtest  involves  the  subject  reading  anything.  He  must  read  and  define 
vocabulary  words  from  a card,  but  simultaneously  the  administrat,  ■ ••■'’'nounci ng 
the  words.  Thus,  the  subject  is  not  solely  dependent  upon  his  ability  to  read.  A 
study  measuring  the  differential  literacy  requirement  of  the  two  tests  appears 
in  order. 

The  second  area  involves  the  tester.  The  WAIS  is  administered  individually 
by  professionally  trained  testers.  The  AQE,  however,  is  a group  administered 

and  scored  by  recruiting  testers  who  have  had  minimal  formal  training  in  testing. 
Recruite  r effects  in  the  distribution  of  scores  are  readily  apparent.  For  example, 
the  clustering  of  scores  just  above  AQE  A1  cutoffs  for  career  field  entry  was  found 
in  this  study  and  others  conducted  by  this  division.  A rescoring  of  the  AQEs 
administered  to  our  subjects,  or  even  better,  the  administration  of  the  AQE  to  a 
selected  sample  of  airmen  in  a situation  more  conducive  to  test  taking  than  a 
recruiter's  office,  may  produce  more  valid  AQE  scores. 

In  conclusion,  our  data  reveals  that  black  Air  Force  recruits  score  significantly 
lower  than  white  recruits  on  the  AFQT,  AQE,  and  WAIS.  What  this  means  in  terms 
of  the  true  abilities  that  blacks  and  whites  are  bringing  into  the  Air  Force  involves 
The  identification  and  control  of  variables  in  these  tests,  such  as  literacy  and 
item  specificity,  that  arc  masking  a valid  measurement  of  ability.  Subsequent 
research  should  explore  th<  test  format,  administrative  environment,  and  item 
pools  of  Air  Force  selection  tests  to  determine  exactly  what  is  being  measured  by 
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these  tests. 


Finally,  we  would  like  to  comment  on  racial  bias.  We  have  purposefully 
couched  our  results  in  terms  of  racial  differences  in  scores  rather  than  explaining 
them  in  terms  of  racial  bias.  Any  interpretation  of  the  cultural  fairness  of  the 
•\QE  based  upon  these  data  must  be  tempered  by  the  fact  that  the  criterion  test, 
the  WAIS,  was  an  experimental  test  for  our  purposes,  rather  than  an  operational 
instrument  in  the  Air  Force  selection  process.  According  to  Thorndike  (1971),  it 
is  the  use  to  which  the  test  is  put,  rather  than  the  test  itself,  which  defines 
the  fairness  of  the  test.  In  other  words,  the  fairness  of  a selection  test  is 
determined  by  its  application  in  the  ^election  process,  not  its  internal  charac- 
teristics, Shore  and  Marion  put  forth  their  conclusions  on  the  fairness  of  the 
AQE  based  upon  its  relationship  to  the  SKT,  an  operational  test  employed  b 
the  Air  Force  to  measure  one  factor  in  promotion.  We  can  only  present  our 
data  in  relation  to  the  WAIS,  which  we  have  utilized  as  a baseline  for  measuring 
general  mental  ability.  Hopefully,  some  clarification  of  the  racial  differences 
in  the  AFQT  and  AQE  will  accrue  from  this  study.  But  we  must  leave  a definitive 
interpretation  of  racial  bias  in  selection  testing  to  other  investigations. 
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J .iblo  1.  Means  and  Standard  Deviations  for  Tests 


Black  Sample  (N 

= 100) 

White  Sample  (N  = 100) 

Test 

Mean 

sp 

Mean 

sp 

t 

Al'QT 

■}  1.  68 

13.  83 

64.  98 

20.  67 

9.  32 

AQE-A 

44.  55 

16.  82 

56.  01 

21.47 

4.  18 

AOE-n 

45.  25 

18.  56 

63.  65 

20.  06 

6.  70 

AQLXl 

48.  26 

16.  72 

60.  90 

19.41 

4.  91 

Agi:-M 

44.  20 

18.  56 

62.  75 

1 8.  95 

6.  96 

VI'^) 

96.  74 

8.  78 

104.  99 

10.47 

6.  02 

I’lQ 

97.  4 1 

9.  65 

109.  30 

10.  57 

8.  26 

rshd 

96.  72 

8.  33 

107.  17 

9.  90 

8.  04 

Inl  ornuition 

8.  87 

2.  08 

10.  77 

2.  34 

6.  03 

Conipiehension 

9.  2 1 

2.  67 

10.  83 

2.  74 

4.  21 

Arithmetic 

8.  68 

2.  09 

10.  70 

2.48 

6.  20 

Siiuil.ivities 

9.  6 1 

2.  39 

10.  46 

2.  59 

2.  39 

Di^it  Span 

19 

2.  76 

9.  67 

2.46 

1.  29 

Yoced  alary 

8.  74 

1.  84 

10.  24 

2.  15 

5.  26 

Dip'it  Symbol 

9.  76 

2.  16 

10.  33 

2.  17 

1.  85 

I’icture  Com- 

9.  69 

1.  75 

1 I.  46 

2.  38 

5.  96 

plot  ion 

Hlock  Oi ni^n 

9.  21 

2.  27 

12.  22 

2.  66 

8.  58 

I'ir  lure 

9.  97 

2.  11 

11.  12 

2.  45 

3.  53 

Arrangement 

i'>bi<.'e t Assem - 

9.  01 

2.  31 

11.  73 

3.  08 

7.  03 

Mv 

With  l9d  dl  t - 1.  98  at  . 05  level. 

t - 2.  61  at  .01  level. 
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Table  2.  Intercorrelations  of  WAIS,  AFQT,  and  AQE  Scores  for  the  Black 
Sample  (N=100) 


WAIS  AQE 


VIQ 

PIQ 

FSIQ 

AFQT 

A 

E 

G 

M 

VIQ 

. 46 

. 88 

. 38 

. 27 

. 26 

. 23 

. 10 

WAIS 

PIQ 

. 83 

. 44 

. 19 

. 26 

. 29 

. 23 

ESIQ 

. 48 

. 30 

. 30 

. 30 

. 19 

AFQT 

. 26 

. 37 

. 36 

. 30 

A 

. 31 

. 62 

. 29 

AQE 

E 

. 55 

.40 

G 

. 63 

M 
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3.  Interc  orrelations  of  WAIS,  AFQT,  and  AQE  Scores  for  the  White 
Sample  (N-  100) 


WAIS 

VIQ  PIQ  fSIQ 

Vhd  .54  .91 

U'lt  PIQ  .84 

rsiQ 
APQT 


AQE 


AFQT 

A 

E 

G 

M 

. 64 

. 60 

. 69 

. 77 

. 45 

. 62 

. 28 

. 60 

. 44 

. 50 

. 71 

. 52 

. 74 

. 71 

. 5J 

. 51 

. 75 

. 64 

. 67 

. 56 

. 71 

. 54 

. 73 

. {>9 

. 56 
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Table  4,  Regression  Design 


Regression  Models 

Model  1:  WAIS  IQ'  = WjB  -i  -t  W3(U  x Sel  Test)  + W4  (W  x Sel  Test) 
Model  2:  WAIS  IQ'  = wjB  + W2W  + W3  (Sel  Test) 

Model  3:  WAIS  IQ'  = w (Sel  Test) 


F - Tests 


(Rf^  - 

(1  - Rj^)/df^ 


Full 

Restricted 

Model 

Model 

Testing  for: 

Analysis  I 

Model  1 

VS 

Model  3 

Racial  Difference 

Analysis  II 

Model  1 

vs 

Model  2 

Slope  Difference 

Analysis  III 

Model  2 

vs 

Model  3 

Intercept  Difference 

Definitions 

WAISIQ  = VIQ,  PIQ,  or  FSIQ  score 

w = Variable  weight 

B = 1 if  black,  0 if  not  black 

W = 1 if  white,  0 if  not  white 

Sel  Test  = AFQT  or  AQE  AI  percentile  score 

2 . . 

Rp  = Squared  multiple  correlation  of  full  model 

2 

R^  = Squared  multiple  correlation  of  restricted  model 
2 

Rj  = Squared  multiple  correlation  of  model  1 

dfj  = Independent  vectors  in  full  model  minus  independent  vectors  in  restricted  model 
df^  = Elements  in  vectors  minus  independent  vectors  in  model  1 
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Table  5.  Summary  of  Regression  Analyses 


1’  tests  for  Presence  of  Racial  Difterence  (slope  or  intercept  differences)^ 


Model  1 Model  3 


Criterion 

Predictor 

K,,-’ 

1 

‘^r“ 

r 

prob 

VIQ 

AFQT 

. 40742 

. 40058 

2 

196 

1.  13 

NS 

VIQ 

AQi:-A 

. 35580 

. 27294 

2 

196 

12.  61 

. 001 

VIQ 

AQF-F 

. 41354 

. 34079 

2 

1"6 

12.  16 

. 001 

VIQ 

AQF-G 

. 46722 

. 3o258 

2 

196 

19.  25 

. 001 

VTQ 

AQF-M 

. 25999 

. 1737P 

2 

196 

1 1.  55 

. 001 

PIQ 

AFQT 

. 4761  ' 

, 44 "30 

2 

1"P 

5.  01- 

. 01 

PIQ 

.\Qi:-A 

. 2999 1 

. 11748 

2 

1"6 

2 5.  54- 

. 00! 

PIQ 

AQi:-n 

. 42403 

.3171" 

2 

l"6 

18.  18 

. 001 

PIQ 

AQE-G 

. 36390 

. 22263 

2 

196 

2 1.  7p* 

. 001 

PIQ 

AQF-M 

. 37527 

. 2p173 

*> 

1"P 

17.  8, 

. 001 

rsiQ 

AFQT 

. 5402! 

. 52540 

2 

196 

3.  16* 

. 05 

FSIQ 

.AQE-A 

. 38758 

. 240P2 

2 

196 

23.  52* 

. 001 

FSIQ 

A(,tE-E 

. 51365 

.41136 

2 

196 

13.  74 

. 001 

FSIQ 

AQE-G 

. 49822 

. 367  Ip 

2 

196 

25.  60 

. 001 

FSIQ 

AQE-M 

. 38220 

. 26353 

2 

196 

18.  82 

. 001 

Note.  Starred  (■'■)  F tests  denote  rare  main  effects  (intercept  differences  but 
no  slope  difference).  All  other  significant  differences  were  shown  to  be  inter- 
action effects  (both  slope  and  intercept  differences). 
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Docs  the  USAF  Officer  Biographical  Inventory  Portion  of  the  AFOQT 
Inadvertently  Measure  Authoritarian  Personality? 


a paper  for  presentation  at  the  15th  Annual  Conference  of  the 
Military  Testing  Association 
28  October  - 2 November,  1973 


by 

Lt  Col  Ctiarles  W.  Haney 
Chief,  College  Curriculum  Division 
Air  Force  Reserve  Officers  Training  Corps 
Maxwell  AFB,  Alabama 


INTRODUCTION 


I . 

In  the  present  all-volunteer  armed  forces  environment, 
it  is  particularly  crucial  to  our  national  interests  that 
VIC  maintain  an  adequate  and  effective  officer  corps. 

While  academies  and  officer  training  schools  provide  sig- 
nificant inputs.  Reserve  Officer  Training  Corps  (ROTC) 
commissioning  programs  provide  the  greatest  single  source 
of  newly  commissioned  officers  for  the  armed  forces. 

ROTC  programs  have  experienced  a significant  decline 
in  enrollments  in  recent  years.  There  are  a number  of 
factors  to  wiiicii  tnis  decline  can  be  attributed;  the  zero- 
draft,  anti-militarism  and  apatny  toward  the  military, 
and  the  relatively  large  scale  eradication  of  compulsory 
ROTC  programs  on  America's  campuses.  In  an  effort  to  off- 
set this  decline,  the  ROTC  programs  have  expended  consid- 
erable efforts  to  maintain  viable  programs  on  our  nation's 
campuses . 

Alternatives  to  campus-based  commissioning  programs 
nave  not  been  particularly  appealing  even  to  many  wlio  iiave 
been  less  than  sympatnetic  toward  the  military.  The  leaven- 
ing input,  via  ROTC,  of  young  officers  fresh  from  the  human- 
izing and  liberalizing  influences  of  tne  college  campus 
environment  tends  to  offset,  according  to  many,  the  relative 
ly  rigid  and  authoritarian  structure  of  the  military  environ 
ment.  (Karsten  et  al,  1971) 
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I I . BACKGROUND 


We  at  Air  Force  ROTC  have  been  concerned  with  providing 
a respectable  campus-based  procurement  source  for  young 
officers  of  liigh  quality.  In  recent  years  the  emphasis  of 
our  program  has  been  placed  on  attracting  and  motivating 
qualified  students  to  pursue  our  pre-commissioning  program. 
We  have  been  particularly  sensitive  to  certain  assertions 
that  ROTC  tends  to  attract  students  with  traits  related  to 
militarism,  authoritarianism,  and  other  behavior  patterns 
which  might  be  considered  contrary  to  the  traditional 
egalitarian  atmosphere  of  the  college  campus.  (Gage,  1964; 
French  and  Ernst,  1955;  Lucas,  1967) 

A point  of  concern  was  whether  or  not  the  Air  Force 
Officer  Qualifying  Test  (AFOQT) , one  of  several  selection 
criteria  for  cadets  seeking  entry  into  the  advanced  AFROTC 
course,  or  seeking  scholarship  status,  might  be  inadvertent- 
ly functioning  to  screen  out  the  more  egalitarian  students. 

One  portion  of  the  test,  the  Officer  Biographical  inven- 
tory (OBI) , became  suspect  as  a result  of  a study  conducted 
by  the  AFROTC  Education  Evaluation  and  Research  Division 
in  the  summer  of  1970.  (Gordon,  1970)  In  this  study  it 
was  found  that  a relatively  low  correlation  existed  between 
grade  point  averages  (GPA's)  and  the  Officer  Quality  Com- 
posite (OQC)  scores  of  the  AFOQT  for  468  cadets  attending 
AFROTC  Field  Training.  Table  I contains  these  correlations. 


TABLE  I 


r 


Correlation  of  OQC  Components  With  GPA 
(Gordon's  Study,  1970)  N = 468 


Correlates 

r 

Overall  OQC 

.22 

Verbal 

.20 

Quantitative 

.20 

Officer  Biographical 
Inventory 

-.06 

The  relatively  low  coefficients  of  correlation  between 
GPA's  and  the  verbal  and  quantitative  portions  of  the  OQC 
notwithstanding,  Gordon  reasoned  that  the  lack  of  correla- 
tion between  GPA  and  the  OBI  reflected  cause  for  concern. 

He  further  analyzed  relationships  between  the  OBI  and  over- 
all OQC,  Verbal  and  Quantitative  components,  GPA,  and  the 
AFROTC  Field  Training  Rating,  Table  2 contains  these  cor- 
relations . 
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TABLE  2 

Correlation  of  OBI  with  Overall  AFOQC , Verbal,  and 
Quantitative  Components,  GPA,  and  AFROTC  Field 
Training  Rating 
(Gordon's  Study,  1970)  N = 468 


Correlates  r 


Overall  AFOQC 

.26 

Verbal 

-.17 

Quantitative 

-.24 

GPA 

-.06 

Field  Training  Rating 

. 14 

These  findings  led  Gordon  to  suspect  that  the  OBI  might 
be  dysfunctional  to  the  overall  OQC . It  might,  for  instance, 
be  detracting  from  the  purported  relationship  between  the 
OQC  and  overall  academic  achievement.  It  might  therefore  be 
detracting  from  the  intended  purpose  of  the  OQC  which  is 
"to  evaluate  aptitudes  which  are  important  for  commissioned 
officer  performance  and  success."  (Air  Force  Manual  35-8, 
1969) 

The  OBI  component  of  the  OQC  is  purported  to  consist  of 
'items  pertaining  to  past  experiences,  preferences,  and 
personality  characteristics  known  to  be  related  to  success 
in  officer  training."  (Miller,  1969)  This  fits  a general 
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description  of  measures  of  attitude.  There  are  many  varieties 
of  attitude,  however,  which  can  be  treated  as  independent 
variables.  The  purported  purpose  of  the  OBI  does  not  spe- 
cifically identify  any  particular  variety  of  attitude.  Fur- 
ther study  was  needed,  therefore,  to  shed  additional  light 
I on  tne  true  function  of  the  OBI.  This  prompted  the  present 

study  to  investigate  the  relationship  between  the  031  and 
known  measures  of  authoritarian  personality. 

Since  other  researchers  had  speculated  that  military 
officer  training  programs  tend  to  attract  those  with  "mili- 
taristic" and  "authoritarian"  traits  of  personality  (hammers, 

1965;  Lucas,  1967),  it  seemed  appropriate  to  choose  a trait 
such  as  "authoritarianism"  for  comparison  to  the  OBI.  It 
was  reasoned  that  useful  information  could  be  obtained, 
through  baseline  research  procedures,  by  correlating  OBI  raw 
scores  with  scores  obtained  from  measures  of  authoritarian 
personality.  If  a significant  relationship  was  found  to 
exist  between  these  sets  of  data,  then  a relationship  might 
be  inferred  to  exist  between  the  OBI  and  measures  of  authori- 
tarian personality. 

III.  PURPOSE  OF  THE  STUDY 

The  purpose  of  this  study  was  to  determine  if  a signifi- 
cant relationshin  could  be  inferred  to  exist  between  the 
Officer  Biographical  Inventory  (OBI)  component  of  the  Officer 
Quality  Composite  (OvC)  of  the  Air  Force  Officer  Qualifying 
Test  (AFOQT)  and  measures  of  authoritarian  personality.  The 
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hypothesized  relationship,  if  confirmed  to  exist,  might  have 
been  the  basis  for  criticism  not  only  of  the  AFOQT  but  of 
the  Air  Force  ROTC  Program  selection  and  commissioning  pro- 
cess which  the  AFOQT  directly  affects. 

IV.  PROCEDURE 

The  California  F-scale  was  selected  as  a suitable  instru- 
ment to  measure  authoritarian  personality  for  the  purpose  of 
this  study.  Despite  some  criticism  of  the  F-scale,  it  has 
been  widely  accepted  as  a valid  measure  of  authoritarian 
traits.  Three  versions  of  the  F-scale  were  adopted  for  use 
in  this  study.  These  were  the  Form  78,  and  Forms  40  and  45. 
These  were  taken  virtually  intact  from  the  pages  of  The 
Authoritarian  Personality,  Adorno  et  al.,  (1950).  An  addi- 
tional scale  was  developed  and  validated  by  the  principal 
investigator  and  Dr.  Earl  Jones,  Texas  A&M  Univ. , to  act  mere- 
ly as  a stabilization  cross-check  of  the  F-scales.  Adorno's 
Form  78  is  referred  to  hereinafter  as  F-1.  Forms  40  and  45 
were  combined  into  one  scale  and  are  referred  to  hereinafter 
as  F-2.  The  special  cross-check  scale  is  referred  to  as  the 
JOHAN  (Jones  and  Haney)  scale. 

The  Air  Force  ROTC  cadet  population  at  Texas  A&M  Univer- 
sity (April,  1971)  was  selected  as  subjects  for  this  study 
because  of  the  relative  availability  of  AFOQT  data  on  this 
group.  Scales  F-1,  F-2,  and  JOHAN  were  administered  to  the 
Texas  A&M  cadets  and  this  yielded  391  usable  samples.  At  the 
time  these  scales  were  administered,  this  number  represented 
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approximately  60  percent  of  the  Texas  A&M  AFROTC  cadets  who 


had  taken  the  AFOQT.  This  was  deemed  a sufficient  number  of 
respondents  to  conduct  a valid  and  significant  study. 

An  analysis  of  F-1  and  F-2  scores  from  the  sample  popu- 
lation revealed  tiiat  tiie  mean  scores  obtained  were  very  close 
to  the  norms  cited  by  Adorno  et  al.,  (1950).  Figure  1 m 

illustrates  this  comparison.  The  F-1  and  F-2  scales  adapted 
to  this  study  used  five  point  Likert-type  scales  while  the 
Adorno  scales  were  seven  point  scales.  Nevertheless  the  com- 
parisons were  quite  favorable. 

FIGURE  1 

Comparison  of  Norms  for  F-Scales 


F-Scalo  Form  78  (3.  71)  F-Scale  Forms  40  & 45  (3.  84) 


OBI  -raw  scores  on  each  of  the  respondents  were  extracted 
from  data  available  at  AFROTC  headquarters  at  Maxwell  AFB, 

The  OBI  mean  for  the  sample  population  was  37.  This  also 
compared  quite  favorably  with  the  Air  Force-wide  OBI  raw 
score  mean  estimated  to  be  36  by  the  Air  Force  Human  Resources 
Laboratory.  These  sets  of  data,  the  OBI  scores,  F-1,  F-2 , 
and  JOHAN  scores,  became  the  basis  of  comparison  for  the  pur- 
pose of  this  study. 

As  a first  step  in  testing  the  relationship  between  the 
various  measures  used  in  this  study,  the  following  null  hypoth- 
eses were  formulated: 

H There  would  be  no  significant  relationship  between 
°1  OBI  raw  scores  and  F-1  raw  scores. 

There  would  be  no  significant  relationship  between 
2 OBI  raw  scores  and  P-2  raw  scores. 

H There  would  be  no  significant  relationsliip  between 
°3  OBI  raw  scores  and  JOHAN  raw  scores. 

H There  would  be  no  significant  relationship  between 
'^4  F-1  raw  scores  and  F-2  raw  scores. 

H There  would  be  no  significant  relationship  between 
°5  F-1  raw  scores  and  JOHAN  raw  scores. 

H^  There  would  bo  no  significant  relationship  between 
6 F-2  raw  scores  and  JOHAN  raw  scores. 

V.  FINDINGS 

The  OBI  raw  scores  were  correlated  with  scores  from  each 
of  the  scales  used  to  measure  authoritarian  traits  and  v;ith  the 
JOHAN  scale.  Pearson  coefficients  were  obtained  as  depicted 
in  Table  3.  The  only  significant  relationships  found  to  exist 
were  those  between  the  F-1,  F-2,  and  JOHAN  scales. 
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TABLE  3 


Coefficients  of  Correlation: 
Entire  Population 


Scale 

OBI 

t 

\ 

F-l  F-2 

i 1 

1 i 

i ! 

JOHAN 

OBI 

1 

1 ' 

i 

-.06  ' -.03 

1 1 

. 06 

F-1 

1 

1 

! .78* 

.46* 

F-2 

i 

1 

1 j 

.61* 

* significant  at  .05 

On  the  basis  of  these  data,  the  investigator  failed  to 

reject  the  following  null  hypotheses:  , and  II^  . 

12  3 

The  investigator  did  reject  il^  , , and  . The  investi- 

gator sought  to  determine  if  this  lack  of  correlation  between 
the  031  and  measures  of  authoritarianism  was  a general  charac- 
teristic of  the  sample  population  or  was  possibly  a function 
of  the  combined  class  groupings.  Realizing,  for  instance, 
that  lower  OBI  scores  would  have  been  screened  out  in  the 
process  of  selecting  cadets  into  the  Professional  Officer 
Course  (POC) , the  advanced  level  course  offered  during  the 
last  two  years  of  AFROTC , and  that  greater  authoritarianism 
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was  proUably  to  be  found  araong  the  lower  classmen,  the 
investigator  decided  that  further  computations  were  neces- 
sary in  order  to  treat  eacli  class  as  a separate  entity. 
Correlation  coefficients  \/ere  therefore  computed  for  each 
class  level  adhering  to  the  same  null  hypothesis  in  each 
case.  Tables  4 through  7 show  the  correlation  coeffi- 
cients obtained.  Again,  in  each  case,  the  investigator 
failed  to  reject  the  first  three  null  hypotheses  and 
rejected  the  last  three.  The  results  were  quite  similar 
to  the  correlation  coefficients  obtained  from  the  combined 
group . 

TABLE  4 

Coefficients  of  Correlation: 

Freshmen 


i 

Scale  OBI  : 

1 

1 

1 

F-1 

! F-2 

1 

JOHAN 

1 

OBI 

-.03 

-.04 

I -.11 

F-1 

.81* 

.43* 

F-2 

i 

, .59* 

1 

JOHAN 

1 

1 

1 

j 

* significant  at  ,05 
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TABLE  5 


Coefficients  of  Correlation: 
Sophomores 


Scale 

OBI 

1 

! 

i 

F-1 

F-2  ' 

1 

i 

JOHAN 

OBI 

! 

-.09 

1 

1 

-.06  j 

1 

F-1 

.82*  1 

. 59* 

F-2 

i 

1 1 

i 1 

.66* 

JOHAN 

! ! 
1 

* significant  at  .05 

TABLE  6 

Coefficients  of  Correlation: 
Juniors 


Scale 

061 

1 

F-1 

F-2 

JOHAN 

r 

1 

OBI 

. 13 

.07 

.07 

F-1 

.78* 

. 57* 

F-2 

.69* 

JOHAN 

— i 

i 

> 

1 

t 

1 

1 

»_ 

significant  at  .05 
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'T’AHLL:  7 

Coefficients  of  Correlation: 
Seniors 


Scale 

OBI 

i 

' F-1 

i 

j 

1 

i F-2 

JOiiAN 

OBI 

1 

1 ' 

.05 

1 

.03 

.13 

F-1 

1 

.77* 

.47* 

JOiiAN 

1 

i 

j i 

.63* 

* significant  at  .05 

Conclusions 

The  principal  conclusions  drawn  from  this  study  are  as  follows: 

1,  The  raw  scores  obtained  from  the  measures  of  authoritari- 
anism used  in  this  study  did  not  correlate  significantly 
to  corresponding  OBI  raw  scores.  This  lack  of  correlation 
was  found  to  exist  when  treating  data  from  the  entire  sam- 
ple population  as  well  as  when  treating  data  from  sepa- 
rate classes  within  the  sample  population. 

2.  The  separate  measures  of  authoritarianism  did  correlate 
significantly  to  one  another  when  treating  data  from  the 
entire  sample  population  as  well  as  when  treating  data  from 
separate  classes  within  the  sample  population. 
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Tlie  first  conclusion  was  drawn  by  the  investigator  after 

iiaving  failed  to  reject  the  null  hypotheses  H , H , and  H , 

1 2 3 

stated  earlier  in  this  study,  at  the  .05  level  of  confidence. 
The  same  conclusion  was  again  drawn  after  having  failed  to 
reioct  these  same  null  liypotheses  at  the  .05  level  of  confi- 
dence when  treating  similar  sets  of  data  for  each  of  the 
separate  classes  witnin  the  sample  population,  i.e.,  the 
freshman,  sopnomore , junior,  and  senior  classes.  The  second 
conclusion  was  drawn  after  iiaving  rejected  the  null  hypotheses 

, and  H , at  the  .05  level  of  confidence,  also  stated 
4 5 6 

earlier  in  this  study.  Again  data  were  treated  for  not  only 
tile  entire  sample  population  but  for  the  separate  classes 
within  the  sample  population. 

From  these  conclusions,  the  investigator  drew  certain 
inferences.  Tlie  first  of  these  was  that  tlie  OBI  could  not  be 
considered  to  be  a measure  of  autliori tarian  personality  traits. 
This  inference  was  based  primarily  on  the  assumed  validity  of 
the  F-scalc  as  a measure  of  authoritarianism,  and  the  lack  of 
correlation  between  raw  scores  of  the  F-scale  and  the  OBI. 

The  investigator  then  inferred  that  if  the  OBI  did  not 
measure  authoritarianism,  it  could  not  be  used  as  a means  by 
which  to  predict  authoritarianism.  It  was  further  inferred 
that  the  OBI  probably  would  not  predict  many  of  the  corre- 
lates of  authoritarianism  purported  to  be  measured  in  some 
degree  by  the  F-scales  such  as  ethnocentrism , prejudice,  and 
anti-semitism  (Adorno  et  al.,  1950). 
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The  investigator-  then  inferred  that  if  the  OBI  neither 
measured  nor  predicted  autiioritarianism , then  the  OBI,  as 
an  instrument  of  measurement  and  prediction,  has  no  rele- 
vance in  any  meaningful  discussion  of  authoritarianism. 

The  same  might  further  be  true  in  any  meaningful  discussion 
of  the  correlates  of  authoritarianism. 

These  inferences  were  not  meant  to  imply  that  the  OBI 
could  not  be  used  to  discuss  some  other  construct  in  a 
meaningful  way.  There  are  a number  of  traits  which  are 
widely  accepted  as  indicative  of  success  in  Air  Force  offi- 
cers. Motivation  and  interest  factors  undoubtedly  play  a 
very  important  role  in  the  success  of  officer  candidates 
in  training  or  active  duty  officers.  Other  traits  include 
general  leadership  qualities  such  as  intelligence,  initia- 
tive, resourcefulness , creativity,  loyalty,  dedication,  and 
patriotism,  to  name  only  a few.  Other  personality  traits, 
as  reflected  by  various  attitude  propensities,  are  also 
undoubtedly  important. 

VI . RECOMMENDATIONS 

This  study  was  not  designed  to,  nor  did  it  address  the 
question  of  the  overall  validity  of  the  OBI.  It  was  not 
meant  to  explore  the  OBI's  ability  to  measure  all  of  the 
various  traits  which  may  be  predictive  of  success  in  offi- 
cer training  and  performance.  Although  there  are  undoubted- 
ly many  measures  of  personality  traits  with  which  the  OBI 
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could  be  favorably  compared,  there  are  few  such  studies 
available  in  the  literature. 

It  is  therefore  recommended  that  further  studies  be 
accomplished  in  order  to  compare  the  OBI  to  other  meas- 
ures of  personality.  This  would  shed  new  light  on  the 
overall  validity  of  the  OBI  as  an  instrument  designed  to 
measure  interests,  preferences,  and  propensities  of  mod- 
ern day  youth. 
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HUMAN  RELATIONS  EVALUATION 

Ms.  A.  M.  KELLEHER 
Personnel  Psychologist 

Ihiited  Stales  Coast  Guard  Training  Center,  Governors  Island,  New  York  10004 

PURPOSE 

ie  sLiidv  the  effect  of  the  Human  Relations  course  participation  on 
the  leadershii'  factors  of  consideration  and  structure  was  the  purpose  of 
Liiis  study. 

INTRODUCTION 

Since  the  four  member  Human  Relations  Team  lias  been  conducting  fifteen 
hour  human  relations  classes  at  TRACEN,  N.  Y.,  two  thousand  nine  hundred  and 
live  TRACEN  personnel , seventy-three  Base  N.  Y.  personnel,  and  one  hundred 
lud  fortv-one  other  CG  personnel  have  received  training  in  this  area.  All 
ranks  and  duties  have  been  included  among  the  participants. 

(Questionnaires  on  the  topics  covered  in  class  which  attempted  to  measure 
very  grossly  the  p.ir t ic i pan ts ' attitudes  were  utilized  throughout  the  six 
month  period  of  June  - December  1972.  These  questionnaires  were  tabulated 
and  summarized  by  the  TRACEN  Psycliologist  and  indicated  certain  trends  within 
I anks  on  beliefs  about  tlie  amount  of  racial  descriminat ion  at  the  TRACEN 
and  in  the  Coast  Gu.ird,  beliefs  about  leadership,  and  beliefs  about  military 
problems/benefits  verses  civilian  prob lems /bene f i t s . Tlie  main  results  were 
tliat  Chiefs  anil  Captains  liad  similar  responses  and  P03s  and  Lieutenant  Junior 
Grades'  respondeil  most  similarly.  It  was  tlieorized  tiiat  a man's  place  on  the 
totem  pole  with  its  commensurate  authority  and  respect  within  the  enlisted 
and  officer  structure  seemed  the  best  way  to  predict  pattern  of  belief, 
rhe  other  main  conclusion  was  that  all  felt  leadership  was  a more  severe 
problem  than  race  and  also  the  answer  to  better  human  relations  witliin  the 
Coast  Guard. 

lli'wever,  since  these  questionnaires  were  short,  hard  to  quantity,  and 
frequently  reworded  as  addit ions /de  let  ions  were  made  to  the  course  the  TRACEN 
I’sychologist  considered  it  tenuous  evidence  for  conclusions  about  Coast 
Guard  personnel.  Also,  the  main  objective,  to  evaluate  the  Human  Relations 
ciiurse  was  not  being  achieved. 

Consideration  of  the  results;  however,  directed  the  command  to  tlie  area 
of  leadership.  This  was  one  topic  consistently  mentioned  by  the  Human 
Ki'latious  participants.  It  was  on  this  topic  that  the  Human  Relations 
cv'urse  effectiveness  could  perhaps  be  evaluated. 

After  a survey  of  the  literature,  the  TRACEN  Psychologist  decided  to 
utilize  the  Leadership  Opinion  Questionnaire  published  by  Science  Research 
Associates,  Inc.  This  questionnaire  has  been  used  to  evaluate  human  relations 
training  (Harris  ft  Fleishman,  1955)  and  it  is  designed  to  measure  two  main 
factors  of  leadership,  consideration  and  structure. 

Consideration  (C).  Reflects  the  extent  to  which  an  individual 
is  likely  to  have  job  relationships  with  his  subordinates 
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characterized  by  mutual  trust,  repect  for  their  ideas, 
consideration  of  their  feelings,  and  a certain  warmth 
between  himself  and  them.  A high  score  is  indicative 
of  a climate  of  good  rapport  and  two-way  communl cat  1 on . 

A low  score  indicates  the  Individual  is  likely  to  be 
more  Impersonal  in  his  relations  with  group  members. 

Structure  (S) . Reflects  tlie  extent  to  which  an  Individual 
is  likely  to  define  and  structure  his  own  role  and  those 
of  his  subordinates  toward  goal  attainment . A high  score 
on  this  dimension  characterizes  individuals  who  play  a very 
active  role  in  directing  group  activities  through  planning, 
communicating,  information,  scheduling,  criticizing,  trying 
out  new  ideas,  and  so  forth.  A low  score  characterizes 
Individuals  who  are  likely  to  be  relatively  inactive  in 
giving  direction  in  these  ways. 

Therefore,  the  problem  had  been  defined,  leadership;  the  measurement  tool 
procured.  Leadership  Opinion  Questionnaire;  and  the  subjects  were  selected, 
30  CG  personnel  at  TRACEN,  GINY  - 15  graduates  of  the  Human  Relations 
course  mathched  by  rank  to  15  non-attendees. 

It  was  theorized  that  Consideration  would  be  the  factor  most  likely 
to  be  affected  by  Human  Relations  Training,  although  the  emphasis  on 
direct  communication  in  the  course  should  also  Influence  the  Structure 
factor. 


METHOD 


The  questionnaire  was  test  run  on  a Human  Relations  class  of  nine 
(Commanders  to  evaluate  its  ease  and  pitfalls  of  administration  and 
tabulation.  Also  the  Commanders  were  asked  to  evaluate  it.  The  question- 
naire was  evaluated  as  a usable  and  appropriate  measure  for  the  Human 
Relations  course  by  the  TRACEN  Psychologist. 

The  Executive  Officer  of  me  TRACEN  chose  the  30  subjects  and  delivered 
a questionnaire  and  coverlet  ter  to  them.  The  questionnaire  was  to  be 
anonymous  and  returned  within  two  days  to  the  XO.  The  only  information 
they  were  to  give  was  whether  they  h.id  attended  Human  Relations  course  or 
not . 

The  recipients  were  matched  by  rank. 


Group  1 (did  not  attend) 

1 Commander 

1 Lieutenant  Commander 
1 Lieutenant 

1 Lieutenant  Junior  Grade 
4 Chief  Warrant  Officers 
7 Chief  Petty  Officers 


Group  2 (attended) 

1 Commander 

1 Lieutenant  Commander 
1 Lieutenant 

1 Lieutenant  Junior  Grade 
4 Chief  Warrant  Officers 
7 Chief  Petty  Officers 


15 


15 
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Twenty-nine  questionnaires  were  returned.  Since  they  were  anonymous, 
the  only  information  available  is  that  the  gentleman  who  did  not  return 
a questionnaire  did  not  attend  the  Human  Relations  course.  The  results 
were  tabulated  into  raw  scores  of  the  factors  of  Consideration  and 
Structure  for  eacli  man.  These  raw  scores  were  tlien  converted  into 
percentile  scores  using  the  Norms  Table  provided  by  the  Science  Research 
Associates  company  for  the  Questionnaire  (Table  1).  The  comparison  norms 
selected  were  the  General  Supervisory  Personnel  norms  since  the  groups 
were  combined  of  different  ranks,  and  more  specific  norms,  e.g.  Executive 
norms  would  be  appropriate  for  only  a few  members  of  the  test  groups. 


RESULTS 

The  mean  scores  indicated  that  the  14  graduates  of  the  course  scored 
higher  on  both  Consideration  (61.57)  and  Structure  (58.07)  than  the  non-atten- 
dees (53  on  onsi deration  and  51.2  on  Structure). 

A t test  was  deemed  most  appropriate  to  measure  the  significance  or 
non-significance  of  these  differences.  Two  t tests  were  run: 

1.  Group  1 versus  Group  2 compared  on  Consideration;  the  t test  yielded 
a value  of  6.47.  Witli  28  degrees  of  freedom,  this  value  is  significant  beyond 
the  .01  level. 

2.  Group  1 versus  Group  2 compared  on  Structure;  the  t test  yielded  a 
value  of  9.54.  Witli  28  degrees  of  freedom,  this  is  significant  beyond  the 
.01  level. 

Comparisons  between  Consideration  and  Structure  within  the  two  groups 
were  also  conducted: 

1.  Group  J,  Consideration  versus  Structure;  the  t test  yielded  a value 
of  1.8.  With  14  degrees  of  freedom,  tills  is  not  significant. 

2.  Group  2,  Consideration  versus  Structure;  the  t test  yielded  a value 

of  3.56.  With  13  degrees  of  freedom,  this  is  significant  beyond  the  .01  level. 
The  Consideration  Score  is  significantly  higher  for  Group  2. 

All  mean  scores  , however,  were  above  the  50th  percentile  (Table  1). 

CONCLUSIONS 


Attendance  at  the  Human  Relations  course  results  in  significantly  higher 
scores  on  leadership  factors  of  Consideration  and  Structure  as  measured  by  the 
Leadership  Opinion  Questionnaire  given  to  two  matched  groups  of  attendees  and 
non-attendees.  The  graduate  scores  higher  than  a man  of  similar  rank  and 
expearience  who  did  not  attend. 

Consideration  appears  to  receive  higher  significant  scores  than  Structure 
among  the  graduates,  and  this  factor  may  characterize  the  Coast  Guard  Human 
Reiations  graduate  leader. 
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RAW  SCORES  AND  PERCENTILE  NORMS  FOR  BOTH  GROUPS 

Group  1 Group  2 

(did  not  attend)  (attended) 


Consideration  Structure  Consideration  Structure 


raw  score 

percentile 

raw 

score  percentile 

raw 

score  percentile 

raw 

score  percentile 

1. 

56 

65 

53 

65 

56 

65 

52 

60 

2. 

55 

60 

A6 

28 

A9 

27 

A7 

31 

3. 

A5 

12 

57 

80 

51 

AO 

A3 

17 

A. 

58 

75 

55 

75 

69 

98 

57 

80 

5. 

62 

90 

AA 

20 

A9 

28 

50 

50 

6. 

A9 

28 

56 

78 

57 

69 

56 

77 

7. 

60 

85 

39 

08 

66 

95 

63 

95 

8. 

55 

60 

5A 

69 

50 

31 

AO 

08 

9. 

50 

31 

A1 

10 

52 

A5 

52 

60 

10. 

63 

93 

50 

50 

60 

90 

55 

75 

11. 

59 

80 

58 

85 

63 

91 

52 

60 

12. 

36 

01 

A7 

31 

A8 

25 

53 

65 

13. 

51 

AO 

AA 

10 

55 

60 

55 

75 

lA. 

55 

60 

60 

90 

71 

99 

52 

60 

15. 

A6 

15 

5A 

69 

X = 53 

X = 51.2 

X = 61.57 

X = 58.07 

n = 15 

n = lA 
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Centr«ll7.  i • r 'tion  System  For  Air  Force  Pilots 

by 

Melvin  3.  Majesty,  Lt  Col,  DSAF 


Pilot  training  Is  the  most  costly  type  of  Individual  training  given  by 
any  of  the  services.  In  fact,  it  la  probably  the  most  expensive  eduoaticnal 
process  in  the  world.  In  recent  years,  the  bill  for  pilot  training  for  the 
military  services  amounted  to  over  two  billion  dollars  a year. 

The  cost  of  pilot  training  depends  upon  which  costs  are  included  in  or 
excluded  (e.g.  aircraft  costs)  from  the  calculation.  The  costs  per  gradu- 
ate from  Undorgrad»iate  Pilot  Training  (UPT)  is  about  $90,000.  The  initial 
flight  training  of  an  Air  Force  pilot  takes  almost  a year.  The  program 
includes  flying  training  (using  T-37  and  T-38  jet  aircraft)  of  sufficient 
scope  to  produce  the  necessary  understanding  of  the  principles  and  profi- 
ciency requirements  associated  with  flying  high  speed  Jet  aircraft.  The  UPT 
program  produces  a graduate  who  is  capable  of  transition  into  any  aircraft 
in  the  Air  Force  inventory.  He  must  now  be  trained  to  fly  an  operational 
aircraft.  Upon  graduation  from  UPT,  the  pilot  is  assigned  to  an  operational 
command  for  training  in  a specific  aircraft.  In  the  operational  flying  train- 
ing program,  e.g.  Combat  Crew  Training,  the  specific  procedures  unique  to  an 
individual  aerospace  system  and  missions  are  emphasized. 

New  Centralized  Program 

Recently  the  Air  Training  Command  completed  a mission  analysis  of  pilct 
training  for  the  1975  through  1990  time  period  (Missicr.  Analysis  Study  Sroup, 
1972).  This  mission  analysis  includ'‘c  tht  interface  of  the  screening  ..r.c 
selection  process  with  the  future  Unriorgracuate  Pilct  Training  (UPT)  : yutcr.. 
Properly  derdgned  screening  and  Ctlection  was  considered  a critical  co;..pcnent 
for  the  future  UPT  sjT.tcm.  It  was  concluded  that  a "ten  percent  attritior 
level  is  a realistic  goal  if  centralized  selection  is  employed  and  training 
improvements  are  implemented." 

The  initial  step  in  the  centralized  selection  program  was  the  establish- 
ment of  a light  plane  (military  version  of  the  Cessna  Kodel  172)  flight 
screening  program  as  part  of  the  officers  training  school.  Officers  wno  nad 
preidcusly  completed  a US/vF  sponsored  light  plane  progrrun  would  be  excusec 
from  the  Flight  Screening  Program  (FS?).  .All  other  pilot  candidates  ■.•:culfl 
complete  FS?  prior  to  tipt. 

Along  with  the  FSP  came  the  central! zeo  Class  I physical  examinauio.i  ano 
a program  to  test  and  validate  various  ground  base  screening  devicas  for 
operational  use.  The  centralized  selection  program  maxes  the  use  of  -ndlvid- 
ually  administered  tests  feasible.  It  is  anticipated  that  a grounn  b-se 
screening  system  will  emerge  tc  replace  or  partially  replace  the  initial 
light  plane  screening  program  (F.SP). 

The  ground  based  devices  ivill  be  usea  to  preaict  succe^^  in  the  iigr.t 
plane  FSP  program  as  well  as  success  in  tns  jet  trainer's  (T-37/T-30).  At 
the  same  time  it  will  be  possible  tc  determine  the  extent  tc  which  the  pgp 
will  predict  success  in  Jet  training. 

The  development,  test  and  validation  of  a new  grcunc  based  pilot  screen- 
ing system  Is  the  first  major  effort  on  pilot  selection  to  be  undertaken  by 
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Air  Force  since  World  War  II.  Over  the  years  the  Air  Force  has  continued 
to  update  and  revise  the  operational  system  and  the  selection  procedures. 
Selective  studies  have  been  accomplished  in  the  area  of  pilot  training. 
However,  this  is  the  first  time  a total  new  system  is  being  Investigated 
as  a complete  entity. 

Selection  Procedure 

Although  there  are  several  sources  of  entry  into  pilot  training,  (Air 
Force  Academy,  AFBCTO,  etc)  the  largest  single  source  of  both  entry  and 
washout  in  the  Undergraduate  Pilot  Training  (UPT)  program  is  School  of 
Military  Sciences,  Officer.  Most  of  the  entries  into  the  officer  training 
school  (School  of  Military  Sciences,  Officer)  come  through  direct  recruit- 
ment. The  procedure  begins  with  an  Interview  In  the  Recruitment  Office  to 
determine  whether  the  candidate  meets  the  minimum  requirements,  e.g.  college 
graduate,  citi/.en,  and  he  must  be  able  to  enter  UPT  prior  to  age  27^.  If  he 
is  a potential  candidate,  he  is  given  the  Air  Force  Officer  Qualifying  Test 
(AFCCT).  Upon  scoring  of  the  AFCQT  a determination  is  made  as  to  whether 
he  is  fully  qualified  or  meets  the  minimum  standards  on  the  AFOQT  (Figure  1). 
A fully  qualified  candidate  completes  the  application  form  and  takes  a 
flight  physical  examination.  These  are  sent  to  the  Flipht  Surgeon's  Office 
for  review.  He  is  then  mailed  a class  entry  card  which  specifies  the  date 
he  is  to  begin  training.  He  must  sign  and  retutvi  this  card  to  the  Air 
Force.  If  his  scores  on  the  AFCQT  are  abo\'e  the  minimum  standards  but  not 
above  the  fully  qualified  standards,  the  procedure  is  a little  different. 

The  application  form  is  first  sent  to  the  Selection  Board.  If  the  Selection 
Board  accepts  the  applicant,  the  flight  physical  examination  is  given  and 
the  results  sent  to  the  Flight  Surgeon's  Office  fcr  review.  A class  entry 
card  is  mailed  to  the  applicant  for  signature. 

Potential  candidates  can  be  lost  anywhere  in  the  selection  process. 
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Selection  Process 


Interview 

college  graduate 

citizen 

age 


AFOQT 


Fully  Qualified 


Above  Minimum 


Application  Form 
Flight  piysical 
Flight  Surgeon  Review 
Mail  Class  Entry  Card 
Applicant  Signs  and  Returns 


Application  Form 
Selection  Foard 
Flight  Physical 
Flight  SiLgeon  Review 
Mail  Class  Entry  Card 
Applicant!  Signs  and  Returns 


Figure  1 
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noncept,  of  Cnorntion 

S^\'orn]  polootion  teohni  ques  vn  IT  oontimic  to  be  upe  rinrlri.''  tbo  vali- 
dation of  rrour.d  based  scrooninq  dovices.  If  the  potential  candidate  does 
not  satisfv  the  rntnirmim  reof' ’'oments  aa  determined  throiuh  the  interviev^ 
he  cannot  apply  for  pilot  training.  Tf  he  doer  not  qualify  on  the  rFC'^.T, 
he  cannot  siirmit  an  application.  If  his  application  has  to  be  acted  upon 
by  !.he  Selection  Poard,  he  rmist  be  favorabl;/  considered  by  the  board  be- 
fore he  can  take  the  flipht  physical.  He  may  be  disqualified  on  the  basis 
of  tbiis  flipht  physical  or  the  Class  7 physical  eyamination  he  receives  the 
first  fev  davs  after  arrival  for  the  training  procram.  ^fter  he  be,T'-'<; 

■■^';ic-o  re  el'mincTPc  it  tne  ^aree-'i-  '■'rcqr' ■'  -r'-r 

is  a siy  to  seven  week  program  of  fourteen  flying  hours.  F.S?  occurs  between  thft 

third  and  foi;rth  training  week  in  the  Schorl  of  Military  Sciences,  b'^'flcer 

/■cypp). 

""hf-  'round  based  screenin'’  "iev'cer  aco  admi ni stored  as  scon  as  pc''''-iole 

officer  trajninp  pro. 'ram  and  b.-’fore  he  -ecins 
pep.  Hovic'e’*,  the  results  cf  the  ■'rcund  h.’.sc-d.  rcreeninc  are  not  available 
to  tne  •'andidate,  the  faculty  or  any  one  involvod  in  tr.e  Taininp  prepra-a. 
tn  fact,  mC'S'^  of  the  qrcund  based  te.nt  a"e  not  even  scored  until  srm.-tiro 
aftfc  'T’Tiletion  of  FSP. 

'•■'mpb '-si  s is  on  the  use  of  j r.su'ouaien  ; t.'Ut  ere  cc  nsidu "e-d  to  be  "off-the- 
sr.l:''  ’“■'>■''0 -.so  the  initial.  de''a';lcp''crt  tlro-aoly  brer,  a.  cc'npl  i "ced . This 

iiiniri  wt  r'ru^  l-’di  xim?  reo- red  to  '■■■  ' e ins'^  'idr'-'r.-  '.pc '"e-  or  ' 

s "•  ' re  “ r-  .'-n 00  "''r;  tudt-  'O  .'(  ■"’'itc'"  ...p-  — . ..y  ^ - : 

r'" ' r:pitude  and  o'"c  o r.ta’"'^  o.  ''ef'.'  ' ret""  n " .. 

Too  rS  ,a  ' i 'ht  pl'n'  cnio'  dc"'.''':  'al!  oo  p-.v,'.'"*''  ' . ' ■ • 

.•."lon  "e  "cu  perrit,  *^asic  Surv''vai  . y.  -q 

■.nq  i'ov-o(r.  '■p'v:  rachr.i  qU'S;-  si '' h-  in':  po;'  '•c  iri'c  the  oper.'  ■ ■ — ^al  1 ■'-'■•-r 

s'ust' .'i  c’.s  scon  a.'  pcs-'ible  afto's  va'!''  d,ity  h'^:.  been  drm'rs-ratori.  Irnr uroont  ly, 
*ht  li  :ii.  plane  t’lyin.-  soreeiun  ■ jir-'  'ram  , .y  hr  o-  duopd,  .'ilteroc  incerpe- 
;-aT(-  u'r.e  -’il'icient  ir-i'lipht  scrco-rini'  tpchr.ioup;. 

Procrac.  ■ 'ic  ‘-.i  er. 

. arr.  coil^cTion  began  at  the  Sc'r.col  ni'  Military'  oo:  fences , ■'Officer  >'  '-■€) 

'■•iih  t.h'  .■■-’'i  '■'as;  ohodulori  to  rt.  > Perooninc  i nr.triirv  nt.-  ar-'' 

both  'r<  up  .'inri  i o.di  viaaally  admi  ni  ; ■‘/•ro  d . “ost  ci  -r.'or  ..r;n'l ni  sT,'.  red 

1 r.  strum -nt.s  are  tim-'d  raerr  and  non.'i  ’ *'  t-,.  ■;  ndi  vi  du '•!  1 >r  'd'^ii  ri  r te 'O'-d 

te;;t  tend  to  'oe  untimod. 

n 'now  of  the  'iifforences  in  'rie  types  of  'no  tett  -vai  i ar  il  i tv 

the  nrcgra.T  vas  orprini  r.ed  into  thr'‘c  -'ns.''es‘ 


■■ha  S'-  1 


,P7"  - "erher  197'- 
'rov  .97’'  . Vr— .cv^ 

107  I ..  7 - 07.- 


-'auc.aticr.  wj  i . . '.n  sr-n  .■t.ap'iet.s  '<'5 '..I- 

idaticn  vnll  coni.in-so  arte  trr-  Jst  pilot  tru-. nq  pro,  ram  (Undorc  'ud..:att 
^ilot  Troinir  ;}.  ^ r y-.^-  ' 'iC  •r.ndu'*i 'cs  the.  last  'lass 

to  craduote  2r:~.  P':\  itit r'f ore , 't  is  ■■ '.t.i  cj ’-sTed  a*,  'ull  - ai  id;. ti'''"  ■.•'i  i 

not  be  completori  rrrt.il  1976. 

"'ne  te'tirc  content  of  eacn  ir*'  shivc"  'n  ’’ifurre  P. 


ar''  shi  viT 


Figure  2 


PHASE  I 


Group  Teate 
Altitude  Chamber 

PHASE  II 

Group  Tests  (Including  Phase  I tests) 
Altitude  Chamber 
Individual  tests 


PHASE  III 

Group  tests  (including  Phase  I and  Phase  II  tests) 

Altitude  Chamber 
Individual  tests 
Selection  Board 

It  is  intended  that  tests  in  any  previous  phase  would  be.  admi nt  stereo 
ir  all  subsequent  phas'=-s.  However,  removal,  substitution  or  recommended 
addition  of  tests  will  be  a function  of  the  validation  program.  The  unioue 
feature  of  Phase  II  Is  the  introduction  of  individually  administered  test- 
ing. Convening  of  a Selection  Board  when  the  student  reports  for  training 
starts  in  Phase  III. 

Figure  3.  PHASE  I 

Air  Force  Officer  Qualifying  Test  (AFOOT) 

Reference  Test  Battery 
Navy  Aviation  Selection  Board 
Strong  Vocational  Interest  Planks 
Altitude  Chamber 
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Figure  3 


(con' t) 


PHA?F.  IT 


Phase  I test 

General  Aviation  Trainer  (GAT-1) 
Complex  Coordination  Test 
Attitude  Survey 
Interviev 


III 

^hase  I and  -^hase  IT  tests 
Aviation  Indoctrination 
Personal  adjustment  Tests 
Selection  Beard 

The  specific  tests  in  each  pnasi-  arc  listed  in  Figure  3.  These  instru- 
ments vpra  selected  as  a result  cf  nrevicus  research  and  demonstrated  poten- 
tial. ’■vT.e.nt.;ver  oossible,  a fully  dtvolr.pca  test  would  he  identified. 

AFOOT ; 

The  A“COT  is  used  in  officer  and  aircrew  selection.  It  is  admi ni  .stered 
aurini  rnt'  recn:itin<j  prcc.cs.’^.  Thcr-.;  ira  five  test  booklets  containinjj 
th’ '"t  "•■■n  ■■  rt,.-  fhcch  yield  so''ro'=*  rrator-Tech’’". 

Of f to."  c 'J'.i.a'' i tj'-.  Verb^'’  •>'-d  f ■.""‘  itof,  v r to'it  T-=li  dated  -in 

1963-19o5  classes.  A more  recent  partial  validation  was  accomplished  with 
a i.  mple  of  19b9  classes. 
liEFr.LFNCb  IbST: 

The  Heference  Test  "attery  is  an  exct.. rimentai  battery  oevelcped  by  tne 
•vir  r'orce  Human  descurces  Labcratcry.  there  are  tweni.y-cne  brief  subtests 
in  four  bcuklets.  It  is  designed  tc  yield  measures  of  several  verbal  and 
nnrr.erbaT  aptitude  factors  including  osychomotor.  '’'hese  instn;ments  will  be 
useful  ii;  interpreting  scores  on  the  oth'^r  tests. 

NA’T  TbST; 

Thf  'I-.v.  Na'vy  and  Marine  Corps  Aviation  Selection  battery  i.n  rsed  for 
5e.leti.ion  cf  pilcts  and  flirht  officers  in  the  N.avy  ar_i  Marine  Corps. 
cf.n..i.‘?to  of  academic  j nn^chanioal.  sp.at.i  "’1  .nr;;  hi  c ranh'' cal  booklet:-. 

INTEREST  TE.FT; 

The  Strong  Vocational  Into  in- at  clai'ii-.  'ieln;  chiroy-ceve.s  interest, 
assessments.  It  is  being  used  in  research  on  career  intent  of  junior  officers 
in  the  Air  Force.  Previous  Navy  research  with  this  instru.T.enc.  has  shewn 
premising  results. 
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ALTITUDE  OHAJIPEP! 

The  Allitude  Chamber  medical  f?valuatior  takps  place  bel'ore  the  third 
trainlPf  veekT  Nr  action  is  taken  on  any  chamber  problems  tin] ere  these 
reappear  later.  The  altitude  chamber  ride  is  scheduled  as  far  as  porsjble 
before  the  ESP  in  order  to  prevent  any  i->hyr.iolo>’lral  difficulties  from  in- 
terferrlnp  with  or  delaying  flying  in  the  F.IP, 

SIMULATCR: 

The  General  Aviatirn  Trainer  ( lAT-1 ) is  an  "off-the-shelf'  light  air- 
craft simulator.  Hev/ever,  development  of  the  test  is  under  contract.  The 
nurpose  of  the  contract  is  to  secure  design  and  full  development  of  a bat- 
tery cf  "leamine  sample"  moa5urec  which  woulfi  serve  as  training  success 
predictors.  This  cont.ract  includes  design  anc  development  of  ttie  equipment 
system  needed  for  administration  o'"  the  battery  as  well  as  develonment  and 
documentation  of  associated  softvsre  (e.g.  cemnuter  procrams  and  syllabus) 
which  provides  the  training  context  in  which  the  measurement  battery  is 
embedded.  Emphasis  is  on  the  problem  of  flying  deficiency  elimination 
(or  failure  due  to  inability  to  develon  appropri ate  flight  behaviors)  in 
UPT.  Tt  is  felt  that  an  objective  sample  of  actual  learning  behavior  in 
the  context  of  pilot  training  using  a lioht  aircraft  simulator  srould 
;/leld  effective  predictors  cf  training  performance  in  the  aircraft.  'incf 
the  test  is  nc\i  under  development,  it  viill  not  he  available  urtil  Phase  TT. 
PPyPHCMOTOR  Tb.ST; 

The  Complex  Coordination  Test  is  the  cutgrcvith  of  previous  research  so 
develop  a new  psyohemeter  test.  Although  still  in  use  for  pilot  selection 
by  several  foreign  countries,  '^sycrc^orcr  es-Mng  ha*-  not  beon  in^lusc  b in 
Air  Force  pilot  selection  test  baiti  ries  since  the  earl;'  l^sCs,  b-e  oi 
psychemotor  tof;ts  was  dl  scent  ir.u>' d by  the,  <ir  ?crc',  short!’'  s’"  - • ' c'Fs  '■'r.u 
TI.  It  was  tcc  orstly  arc  difficult  to  maintain  ann  calilretc  ’..n.-  rrary 
separate  pieces  of  ccuinment  under  decentrali  2>:'d  t<^stir  j ^oi  bitarrs. 
preliminary  investi  gaticn  with  the  nevlv  develcpeh  psychcr.s  * or  te^t  i'ld-l- 
cated  that  the  psychemotor  test  is  stiV>  a ■"■r'’dictcr  of  success  In  p'lot 
training. 

ATTITI'DF  SmiVT,Y; 

The  .'.ttitiiae  .''urve}'  is  an  inver  tory  of  porfsnality  ..r.c  r.acKj  i '. r.c 
characteristics  rele'-ant  to  officer  car  er  intent,  "his  cue stiennaire  is 
the  outgrowth  of  nrovlous  work  in  the  area  of  career  insert. 

INTERVITO: 

An  Intt-rvaiw  with  a nsgebn  at  ri  st  t<  rtr:  t.bi’  oandio.' t f ; rt'iru  •.  i ii.g' 

for  military  a<?rcn.at;ti  cs  ( r^'-zyr. '■  ^ supp'ie'TK'T:  cd  ’ / inierviei  J.-t..  g-.s-hi. 
- information)  obtained  by  C'aestionnaire.  '‘revious  research  nas  or.cwn  shat 
tervlew  data  o'otained  by  ouestienpai re  could  be  used  te  identif’'  nilfi-  train- 
ing dropouts  (.snlf-initiat(-d  olimiris.t-.  or.sl, 

AVIATION  TNjK.CTIiIWATTON: 

An  estimated  three  hem-  orient  ot Ion  per',  on  'nili  be' u.'ec  to  • . iot 

candidates  realistic  information  or  allot  training  and  the  rilot  c=;vtr  :n 
the  Air  Force,  This  orientatic.r  i.'  nsv.-  under  ievelop.-tf  nt . Tt  is  ar.s.i''ip^rf-r 
that  a multimedia  'nrieflng  will  be  nreparec  to  l‘"t  candidates  '.crov;  is 

is  like  to  he  an  operational  pi'ot  -'-.bp  -n  isses'"  -nt^  ••■iT  -o  - 4- 
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in  con^-'unciicn  with  this  briefing. 

PFitSOfiAL  A1).)USTK!':OTj 

There  tests  will  address  the  affective  domain.  Emphasis  will  be  on 
inentl  t'vlnp;  the  fear  of  flying,  manlfestat  Ion  of  apprehension  and  motivation 

eHmirvitions. 

PF.T^-m'.TK  h HOARD: 

A S«Teetlon  Board  will  review  the  candidate's  records  and  make  a deci- 
sion nr  to  wtiether  or  not  he  is  a good  risk  as  a pilot  candidate.  In 
.ni.i1 1 i on,  the  oanfildate  may  be  required  to  meet  with  the  Foard. 

Val  idat.i  on 

thi  s Ts  a longitudinal  study  t nvol’-rlng  predictive  validity.  Testing 
IS  done  over  a two  dav  period.  There  will  be  approximately  one  hundred 
q,'ore‘  'ind  .sut'roon-s  on  the  various  selection  Instruments.  Tests  will  be 
oombl nod  to  oom-'ute  the  best  total  prediction.  Prediction  wll]  be  made  for 
attrition  and  the  attrition  categories.  Predictions  will  be  referenced  to 
tr.e  pilot  tr<dning  aircraft  phases.  Consequently  validation  for  each  class 
cannot  ho  completed  until  graduation  from  DPT.  On  the  other  hand  validation 
lo.'iris  as  si  on  as  the  clasr  completes  the  light  plane  Flight  Screening  Pro- 
gram. Validation  of  each  screening  Instn^ment  continues  until  the  validity 
has  be-n  Cistabt  ish ed. 

Hi  let  soreenirig  effectiveness  will  be  determined  and  screening  alter- 
natives evaluated.  Fased  on  the  results  of  the  validation  program,  reoom- 
inerriatlons  will  he  made  to  change  pilot  screening  methods  and  selection 
crit.  r1  used  In  the  operational  system  for  selecting  students  tc  enter 
t:ie  pile*  training  program.  The  goal  is  tc  develop  a new  screening  and 
selection  system  that  will  contribute  to  brinolno  let  pilot  training  attrition 
detn  to  -^en  percent  or  lower. 
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U.S.  ARMY  PILOT  ASSESSMENT  CENTER  PROJECT 


Colonel  i'/allane  F.  Veaudry 
United  Stanes  Ari.ij'-  Infantry  School 

To  clarify  a point  at  the  outset,  despite  the  title,  this  is  not  a 
project  to  assess  pilots.  It  is  a pilot  project  to  see  if  the  Assess- 
ment Center  concept  has  applicability  to  the  Army's  needs. 

This  paper  v/ill  be  presented  in  the  following  order:  (Slide  1) 

- historical  uses  of  the  Assessment  Center  concept. 

- lUckni ound  of  events  leadlnq  up  to  the  establishment  of  the 
Army  Assessment  Center. 


- The  Ass<.‘ssiuent  Center;  mission,  nryunizution,  truininq  proce- 
dures, development  of  nuti;rials , cxi'rcises  and  scales.  Protest 
te.-hniques , si'lection  of  personne! , schedule.'  of  tests  and  typical 
Sequence  of  events. 

Before  addressing  the  first  topic,  let  me  present  ^definition  of  an 
assessment  process.  (Slide  2) 

"i  !o.;,duri‘  in  whlcl}  indlvidu.ils  par  tlci  patt'  in  various  standardized 
qroup  and  individual  exercises  desiqiu-d  to  elicit  specific  skills 
er  !'eh.,\’iorJ.  Thesi'  skill:;  or  In  haviors  are  ohservi'd  and  scored 
ill/  traineil  assessors"  . 

For  example,  if  it  is  to  be  used  to  measure  potential  for  greater  re- 
stjons i b 1 1 i t y or  advancement  to  a next  higher  level,  those  skills  required 
for  future  assignments  will  be  measured. 

The  Assessment  Center  concept  is  not  new,  although  industry  has  recently 
shown  a great  interest  in  this  technique,  as  one  of  several,  to  better 
manage  [le rsoime 1 . Some  reports  indicate  that  the  German  Army  used 
assessnumt  .echniques  as  early  as  191 1-  During  WWII,  the  Office  of 
Strategii  Services  (OSS)  used  this  technique  as  part  of  their  screening 
and  selection  process.  The  British  Army  uses  the  Assessment  process 
as  the  criterion  for  enti'y  into  its  officer  producing  military  schools. 
The  British  Civil  Service  Selection  Board  was  the  first  civilian  govern- 
ment agency  to  use  the  assessment  technique  and  it  still  uses  the 
procednn..'  today.  In  the  United  States,  American  Telephone  and  Telegraph 
(ATTT)  Company  was  the  forerunner  in  the  use  of  an  Assessment  Center 
in  industry.  Today,  as  many  of  you  know,  AT&T  is  a leader  in  the  use 
of  Assess:. ifii t Centers.  Some  other  major  business  organizations  using 
the  technique  are  IBM,  J.  C.  Penney,  Standard  Oil,  General  Electric, 

Ford  and  Kodak.  US  Government  Agencies  cut  rent ly  using  Assessment 
Centers  or  similar  techniques  are  US  Forestry  Service,  Internal  Revenue 
Service  and  the  Federal  Aviation  Administration.  The  US  Post  Office 
uses  an  Assessment  Center  program  developed  by  the  A:iierican  Management 
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Association.  In  addition  a number  of  State  and  City  Government 
agencies  are  using  the  process  as  are  some  school  systems. 

Now  let  us  take  a brief  look  at  some  events  that  led  to  the  establish- 
ment of  the  project.  I should  not  have  to  review  for  this  audience 
the  social  turbulence  of  the  60s  and  its  impact  on  our  major  institu- 
tions, colleges,  churches  atid  even  big  business.  The  rapid  growth 
of  technology  also  had  its  impact  on  the  Army.  Mechanized  personnel 
records  and  pay  systems,  systems  analysis  and  OR  approaches  to  problem 
solving,  the  search  for  automated  battle  fields  and  new  weapons  systems 
all  were  great  strides  in  efficiency,  but  at  the  cost  of  less  attention 
to  our  "people"  needs.  An  Army  War  College  study  in  I969  found  that 
the  individual  at  all  grades  perceived  the  leadership  climate  to  be 
unfair.  By  direction  of  the  then  Chief  of  Staff,  General  Westmoreland, 
a series  of  actions  was  taken  to  provide  for  mission  accomplishment 
with  maximum  concern  for  the  individual  as  opposed  to  mission  accom- 
plishment at  any  price. 

The  US  Army  War  College  study Leadership  for  the  70s recommended 

as  one  of  its  solution  concepts  to  the  problems  it  identified,  use  of 
contemporary  scientific  approaches  to  leadership  instruction.  The 
CONARC  Leadership  Board  in  its  report  "Leadership  for  Professionals" 
also  stressed  the  need  for  updating  leadership  instruction  in  service 
schools  through  greater  use  of  recent  advances  in  the  Behavioral  Sciences. 
As  the  Vietnam  War  was  coming  to  an  end  and  an  all  Volunteer  Army  started 
to  emerge,  the  impact  on  the  Army  had  to  be  recognized.  A smaller  Army 
had  to  be  more  efficient  and  more  effective.  A Volunteer  Arn)y  had  to 
place  greater  emphasis  on  leadership  at  all  levels  if  it  was  to  be 
successful.  The  unending  flovj  of  "new  blood"  as  needed,  no  longer 
existed.  Quality  became  the  'by-word'  as  quantity  rapidly  diminished. 

By  direction  of  the  Army  Chief  of  Staff,  an  Assessment  Center  was  estab- 
lished at  Fort  Benning,  Georgia.  Initially  the  Center  came  under  the 
guidance  and  direction  of  the  Office  of  the  Special  Assistant  for 
Training  (OSAT)  in  the  Office  of  the  Chief  of  Staff  while  assigned  to 
the  US  Army  Infantry  School.  At  present  guidance  is  received  from  the 
Office  of  the  Deputy  Chief  of  Staff  for  Personnel  through  the  Director 
of  Human  Resources.  In  November  1972  the  first  officer  and  one  NCO  were 
assigned  to  the  Assessment  Center  and  directed  to  find  a building  and 
get  organized.  By  February  the  majority  of  the  staff  had  reported  and 
in  March  the  present  director  arrived  to  manage  the  Center. 

The  mission  of  the  Center  is  as  follows:  (Slide  3) 

a.  Test,  the  assessment  concept  as  a precommission  screening  device. 

b.  Test  the  assessment  concept  as  a Icadcrshi p and  career  develop- 
ment technique. 

c.  Determine  the  feasibility  and  applicability  for  Army  wide  use 
of  the  concept  and  techniques  developed  under  a and  b. 


In  order  Lo  accomplish  the  mission,  the  Assessment  Center  is  organized 
as  fol lows : (Slide  A) 


assl:ssme.nt  center 


HL'AIK>I’ARTERS 

SECTION 


1 

INFANTRY  SCHOOL 

DCSPER/DA 

I 1 

1 

1 

1 ASSESSMENT 

1 

CENTER  1''  1 

1 

ARMY  RESEARCH 

i 

INSTITUTE  1 

OFF  14 

EM  7 

3 t:s  (PSY) 

I 


ASSESSMENT 

SECTION 


COMPUTER 

SECTION 


OEE  2 EM  I 


OFF  11  EM  4 


OFF  1 EM  2 


The  director  reports  to  the  Assistant  Commandant  of  the  Infantry  School. 
ODCSPER/DA  provides  guidance  and  is  the  Department  of  the  Army  point  of 
contact.  Tlie  Army  Research  Institute  provides  technical  advice  and 
support  through  a three-man  field  team  of  professional  research  psy- 
chnloi|ists  co-located  with  the  Center.  The  remainder  of  the  organization, 
Center  headquarters,  assessment  section  and  computer  section  are  self- 
explanatory.  Computer  hardware  support  is  provided  by  the  Management 
Information  Systems  Office  (a  computer  center)  at  Fort  Benning. 

The  total  staff  of  the  Center  consists  of  fourteen  officers,  seven  en- 
listed men  and  three  professional  civilian  psychologists  and  three 
secretaries  I rom  Army  Research  Institute  (ARI).  The  majority  of 
military  personnel  (eleven  officersand  four  NCOs)  are  assigned  to  the 
Assessment  Section  as  assessors  with  nine  of  these  officers  and  two  NCOs 
also  acting  as  counselors.  With  one  exception,  all  of  the  officers  have 
master  's  degrees  in  the  social  or  behavioral  sciences  and  one  officer 
has  a Ph.D.  in  psychology.  These  officers  were  selected  by  name  through 
OSAT/DA.  The  noncommissioned  officers  were  selected  based  on  experience 
and  special  abilities  or  qualifications.  They  are  an  exceptional  group. 

The  tiiie  schedule  established  (or  t tu  assessment  project  is  shown  on 
this  next  slide.  (Slide  5) 
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a.  t'cb  73-30  Jun  73  - organization,  training  of  pcrsonnol , 
development  and  pretest  of  techniques,  tests,  situations 
and  measurement  scales,  establishment  of  program. 

b.  Jul  73-Jun  74  - assessment  and  test  of  techniques  and 
devi ces. 


c.  Jul  74-Jun  76  - follow-on  study  to  determine  validity  of 
concept,  situations , tests  and  techniques. 

One  of  the  first  steps  in  developing  the  test  materials  was  to  determine 
the  behaviors  that  were  to  be  elicited  and  measured.  A review  of  the 
literature  including  the  research  at  Ohio  University  and  the  Army  War 
College  resulted  in  a list  of  dimensions  for  leadership  and  establish- 
ment of  a series  of  gross  indicators  for  these  major  dimensions. 

Examples  are  shown  on  this  slide.  (Slide  6) 


Dimension 

Communication  Skills 
Forcefulness 

Decision  Making 

A total  of  12  broad  categories  or  dimensions  are  used,  as  shown  on  this 
si ide.  (Sl ide  7) 

1.  Social  Skills  (Interpersonal  Competence) 

2.  Communications  Skills 

3.  Adaptability 

4.  Motivation 

5 . Forcefu Iness 

6.  Mental  Ability 


Gross  Indicator 

informal  and  formal  oral 
written  communications 
understanding 

self-confidence 
energy  level 
risk  taking 
ini tiati ve 

use  of  available  information 
decision  quality 
decisiveness 


7.  Decision  Making 

8.  Administrative  Skills 
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. t:ft'(j<jt.ivvnvss  in  Ori^nnizationcil  Loadership  Role 

10.  ^hiporvisorij  .‘Skills 

11.  I’hijsical  Fitness 

I . Tochni'.-al  and  Tactical  Compidence 

As  the  exercises  v-/ere  developed  and  tested,  specific  indicators  were 
designed  and  scales  developed  for  each  of  the  situations.  For  example, 
under  effectiveness  in  interpersonal  situations  the  following  specific 
indicators  are  used:  (Slide  8) 

Fffcctjvonoss  in  Intorporsonal  Situations 

PfiiK>nst rates  skill  in  handling  interpersonal  situations 

Interacts  with  individuals 

In)  acts  with  group 

Assesses  group  norms 

Actively  participates 

Aacepti-d  by  otht-rs 

Avoids  arousing  hosti litg 

As  each  exercise  or  situation  was  developed,  a series  of  test  runs  was 
conducted  using  personnel  of  the  appropriate  grade  and  experience  from 
those  assigned  to  the  Infantry  School,  both  students  and  faculty. 
Adjustments  in  instructions,  complexity,  and  measurement  scales  were 
made  until  we  v/ere  satisfied  that  ve  had  a challenging  problem  that 
elicited  the  behaviors  we  wanted  and  that  these  behaviors  could  be 
a()propriately  scored.  Outside  assistance  was  obtained  for  some  problems 
or  situations.  The  HumRRO  element  at  Fort  Banning  developed  three 
simulates  vdiich  I'll  discuss  later.  The  Cong lonterate  Game  was  purchased 
from  B'/ham-Bray  Associates,  and  selected  standard  psychological  tests 
were  purchased. 

The  t'/iies  of  exercises  used  by  the  Center  are  shown  on  this  slide. 

(SI ide  9) 

Paper  and  pencil  tests 
Interv i ews 

r.e<ider  l(?ss  group  flisciissinn 
War  ijiimt' 


In-Basket 
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Cnmlhit  or  awor^rt'ttcn  ::iruiLHi' 


Wri  t.i  IK/  oxorc  i :/o 

Conglomerate  (management  exercise) 


I'leld  exorcise 


Tlie  Paper  and  Pencil  Tests  <ire  a battery  of  psycho  1 oq  i ca  I tests  which 
measure  intellect,  leadersliip,  vocational  interi'St  and  reading  ability. 

Tiic  Interview  (two  types)  - on(>  is  a background  in|t*rview  to  determine 
motivation,  experience,  problem  areas,  and  the  assessee's  knowledge  of 
his  strengths  and  weaknesses.  The  second,  an  Appraisal  Interview,  is  a 
training  exercise  where  each  assessee  inte-views  two  others  to  select 
one  for  a position  within  a battalion.  This  inteiview  mea'-.e^s  oigani- 
zat  ion  of  th<jught,  communication  skills  and  social  interaction. 

loaderless  Group  Discussion  - consists  of  different  problems  fni  each 
category  of  assessees.  For  Senior  NCOs  it  is  a co'sbined  individual 
and  group  task  where  six  assessees  are  assigned  a mission  of  getting 
a soldier  Iron  their  unit  selected  as  So  I d ier-of - t he-Mont h and  the 
group  being  assigned  a missioti  of  selection  of  one  soldier  from  the 
six.  This  exercise  measures  leadership,  f o* ce I u I ness , persuasiveness, 
organizational  ability  and  group  interaction. 

War  Game  - is  for  officers  only.  This  is  an  assigned  role  rotating 
leader  exercise  conducted  in  two  2-hour  increments.  Teams  of  six 
players  engage  in  cost  effective  analysis  in  a military  force  planning 
environment.  Total  costs,  R£D,  intelligence  ac<)u i s i t i on , balanced 
offensive/defensive  forces  are  all  considered  under  limited  budget  and 
time  constraints.  This  exercise  measures  organ i zat iona 1 and  leadership 
behavior . 

In-Basket  Exercise  - a typical  ISG,  company  or  battalion  commander  in- 
box is  presented  to  the  assessee  who  has  3 hours  to  address  each  item 
in  the  in-box.  This  exercise  measures  problem  solving,  decision  making, 
work  organization  and  leadc-rship.  It  is  followed  by  an  intervievt  to 
discuss  reasons  for  action  taken  and  the  relationship  of  some  of  the 
actions. 

Simulate  - consists  of  a different  exercise  for  each  category  of 
assessee.  NCOs  act  as  platoon  leaders  of  a combat  patrol  with  a mission 
change  requiring  an  air  mobile  operation.  It  is  a 5"hour  exercise  using 
radios  as  the  only  means  of  communication.  It  measures  stress  tolerance, 
risk  taking  and  leadership.  Senior  company  grade  officers  play  the 
role  of  Company  Commanders  in  a similar  exercise.  The  officer  candi- 
dates and  junior  officers  play  similar  roles  in  a civil  emergency 
setting. 

Wr i t i ng  Exerc i se . There  are  two  forms  of  this  exercise.  The  first  form 
requires  the  assessee  to  write  a statement  recommending  discharge  of  one 
of  his  subordinates.  The  second  form  is  a staff  action  paper.  Both 

745 


exercises  measure  the  assessee's  ability  to  marshall  necessary  facts 
from  an  ambiyious  situation  and  relate  them  in  a concise,  complete 
and  grammatically  correct  fashion. 

Conglomerate  - a management  game  used  to  measure  emergent  leadership, 
aggressiveness  and  social  interaction. 

Field  Exercise  - a 5”hour,  leader  designated,  exercise  involving  a 
team  of  six  assessees.  There  are  six  lanes  with  a mission  and  an 
obstacle  in  each  lane.  A choice  of  materials  is  provided.  It  mea- 
sures leadership,  planning,  organization  and  stamina.  Through  the 
design  of  the  exercises  and  by  means  of  questionnaires,  interviews  and 
peer  ratings,  a tri-focal  view  is  obtained  on  each  individual.  His  own 
evaluation  of  himself--his  peer  ratings--and  the  evaluation  of  the 
assessors . 

A poem  that  I saw  in  a magazine  not  long  ago  seems  to  sum  up  what  we 
are  after.  I do  not  knov-j  the  author. 

And  there  were  three  men  v;ent  down 
The  road  as  down  the  road  went  he. 

The  man  they  saw 

The  man  he  was  

and 

The  man  he  wanted  to  be. 

Concurrently  with  the  development  of  the  exercises,  training  was  con- 
ducted for  the  assessor  staff.  Outside  assistance  was  obtained  from  a 
civilian  firm  (Development  Dimensions,  Inc.)  that  conducted  a three-day 
training  session  at  the  Center.  In  addition  selected  professional 
coOnselors,  to  include  Baker  Consulting  Associates,  presented  instruction 
to  the  entire  staff.  In-house  training  consisted  of  each  member  assist- 
ing in  the  development  of  the  exercises,  taking  the  tests  or  participating 
in  the  exercises  and  studying  the  scales.  Then  a series  of  training 
exercises  was  held  using  local  personnel  as  assessees  with  the  center 
staff  conducting  the  assessment. 

Since  we  are  assessing  five  different  categories  of  personnel,  the  exer- 
cises are  prepared  with  different  levels  of  complexity.  Problems  are 
designed  to  "stretch  the  minds  and  abilities"  and  are  cast  in  areas  that 
the  individuals  should  encounter  in  their  future  assignments  rather  than 
in  their  present  experience. 

Assessees  are  divided  into  three  groups  of  six  for  team  exercises  and  to 
provide  flexibility  in  conducting  the  exercises  within  the  capabilities 
of  the  thirteen-man  assessor  staff. 

This  slide  shovjs  the  categories  of  individuals  to  be  assessed  and  a 
schedule  of  assessment  for  the  year.  (Slide  10) 
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ASSESSMENT  POPULATION 
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A total  of  eighteen  individuals  are  randomly  selected  from  Department 
of  the  Army  lists  of  personnel  assigned  to  attend  the  several  courses 
at  the  US  Army  Infantry  School.  The  orders  for  these  personnel  are  anno- 
tated to  indicate  participation  in  the  Assessment  Center  pilot  program 
for  a four-day  period.  They  are  ordered  to  report  to  Fort  Benning  r 'e 
week  in  advance  of  the  start  of  their  regular  course. 

A typical  sequence  of  events  at  the  Assessment  Center  is  as  follows: 

(SI ide  11) 

Three  days  prior  to  assessment  - assessees  report  to  Fort  Benninq 
for  i n-processinq  at  their  student  company. 

1st  day  - assessees  report  to  the  Assessment  Center  at  ISOO  hours 
for  in-processinq  orientation  and  to  heqin  assessment 
exerci scs . 

2nd  and  3rd  day  ~ assessees  undergo  assessment  from  0700  through 
2100  hours. 

4th  day  - 0700-1700  hours,  assessees  complete  'exercises  at  1700. 

They  fill  out  an  exit  questionnaire  and  arc  released 
at  1730  hours. 

(DCS  candidates  begin  their  assessment  days  at  0530  with  an  hour  of 
physical  training.) 

The  facilities  of  the  Assessnient  Center,  other  than  office  space,  con- 
sist of  ten  small  a i r-cond i t ioned  assessment  rooms,  large  enough  for  a 
six-man  assessee  team  with  three  observers,  a large  conference  room 
where  all  eighteen  assessees  can  be  assembled,  and  a video  monitor 
cubicle  containing  six  TV  monitors.  In  addition,  a field  area  with 
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i X lanes,  each  containing  a different  obstacle,  was  constructed. 

There  is  billeting  for  eighteen  assessees  and  a small  lounge  for  relaxa- 
tion as  time  permits.  We  have  the  capability  of  video  taping  in  all 
assessment  i'ooms  with  remote  TV  cameras  and  video  cassette  recorders. 

These  tapes  prove  invaluable  in  the  counseling  sessions. 

For  those  personnel  being  assessed  for  Leadership  Development,  a de- 
tailei.1,  o!)e-on-one  counseling  sessiori  is  provided  within  a week  to 
ten  days  lol lowing  the  three-day  assessment  cycle.  The  Infantry  School 
class  and  our  assessment  schedules  dictate  availability  of  both  assessees 
and  counselors  for  scheduling  of  these  sessions.  The  rating  scales  are 
all  designed  for  keypunch  and  a compi/ 1 er i zed  profile  is  provided  to  the 
counselor  on  each  man  he  is  to  counsel.  In  addition,  gualitative  data 
is  provided  in  reports  from  the  assessors  and  from  the  entry  interview. 

The  counselor  then  prepares  a synopsis  sheet  on  his  assessee  covering 
relative  strengths  and  weaknesses  in  the  areas  measured.  By  careful 
editing  and  clipping  of  the  video  tapes,  examples  are  selected  from 
the  actual  exercises  to  assist  in  the  counseling  sessions.  These  sessions 
art'  open  tnidod , but  usually  last  from  3 to  h hours  to  each  assessee. 

What  has  been  otjr  experience  to  date?  As  seen  on  Slide  10,  "the  assess- 
ment population",  we  have  processed  three  groups  of  captains,  two  groups 
of  lieutenants,  twit  Senior  NCO  groups  and  one  officer  candidate  group 
through  the  assessment  cycle.  We  won't  get  our  first  look  at  ROTC 
students  utit  i I December. 

It  is  too  early  to  talk  in  definite  terms  of  results,  but  I can  speak  of 
t retids  and  early  indications  concerning  the  Assessment  Center  as  a leader- 
shii)  developmetit  technigue.  You  recall  that  Senior  NCOs  and  captains 
entouti-  to  the  Career  Course,  and  lieutenants  enroute  to  the  Basic 
Intently  Officer  Course  all  received  counseling  feedback  based  on  the 
assessment.  In  addition,  each  assessee  completed  a questionnaire  at  the 
end  ot  the  assessment  and  upon  completion  of  his  feedback  session.  With- 
out exception,  every  NCO  and  officer  founrl  the  experience  to  be  both 
challenging  and  rewarding.  But  more  importantly,  they  all  commented 
positively  about  the  training  value  of  some  of  the  exercises.  All  be- 
lieved that  they  had  a better  understanding  of  themselves  and  in  almost 
all  cases  readily  accepted  the  evaluation  of  strengths  and  weaknesses 
presented  by  our  counselors.  A few  disagreed  on  minor  portions  of  our 
f i nd i ngs . 

Let  me  digress  a moment  and  discuss  the  counseling  feedback  technique. 
Ideally,  the  counselor  elicited  from  the  individual  what  he  learned 
about  hi’iselt  dui  i nq  the  assessment  cycle--what  each  felt  his  own 
strengtfis  oi  weaknesses  were.  The  counselor  then  reenforced  those  that 
our  fin. lings  agreed  with  and  discussed  in  greater  detail  any  areas  not 
volunteered  by  the  assessee.  As  I mentioned  previously,  video  tape  is 
used  to  derwn '■  t ra te  specific  areas  of  strengths  ot  weaknesses  when 
appropriate  behaviors  were  recorded  during  the  actual  exercises. 

As  for  prediction  of  success  as  an  officer  candidate  or  for  commission- 
ing, with  only  one  group  tested  it's  too  earl-)  to  comment.  We  are  getting 
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the  behaviors  we  hoped  to  elicit.  It  is  easy  to  discriminate  among 
individuals.  Our  scales  appear  to  provide  good  measures,  but  v-je  do 
not  have  sufficient  data  to  go  beyond  these  general  statements. 

However,  by  the  end  of  the  test  period  we  expect  to  be  able  to  recommend 
the  best  use  of  the  Assessment  Center  technique  for  the  Army.  We  will 
have  sufficient  data  to  address  specifics  in  terms  of  staffing,  organi- 
zation, types  of  exercises  that  lend  themselves  to  general  use  and 
essential  qualifications  or  training  of  assessor  personnel.  As  part 
of  our  experimental  design,  we  hope  to  be  able  to  run  serveral  groups 
of  individuals  without  behavioral  science  backgrounds  or  degrees 
through  the  Center  as  assessors.  With  minimal  training  on  the  exercises 
and  the  scales  as  well  as  some  hints  on  observing  behavior,  we  hope  to 
be  able  to  prove  it  doesn't  take  a master's  degree  to  become  an  effective 
assessor.  We  will  make  this  test  by  comparing  results  between  our 
trained  assessors  and  those  individuals  we  use  on  an  exper i nient a 1 basis. 
One  rather  obvious  fall-out  from  the  experience  to  date  is  the  invalua- 
ble training  the  assessors  are  receiving  from  their  participation  in  the 
program.  Tliere  is  no  doubt  in  my  mind  that  these  officers  and  NCOs  will 
find  that  the  experience  of  close  observation  of  behaviors,  detailed 
study  of  individuals  strengths  and  weaknesses  and  the  practice  of 
counseling  techniques  will  not  only  enhance  their  ability  in  any  future 
assignment,  but  make  them  better  individuals,  too.  You  can't  study 
other  people  without  doing  a little  introspection  and  self-analysis. 

Finally,  there  will  be  a two-year  follovj-up  of  our  results  from  July  Ik 
to  July  76.  This  research  will  be  conducted  by  the  Army  Research  In- 
stitute Field  Unit  at  Fort  Benning.  Through  analysis  of  the  data  we 
will  have  generated  on  individuals  and  groups  as  well  as  through  study 
of  records,  questionnaires,  peer  ratings  and  possible  interviews,  ARI 
will  determine  if  our  results  and  findings  are  valid. 

Gentlemen,  I could  go  on  for  another  hour  about  this  exciting  project, 
but  I know  there  are  many  other  topics  equally  as  exciting  to  their 
sponsors.  I will  be  glad  to  answer  any  questions  at  this  time  or  later 
during  breaks. 


Assessment  Centers:  Validation  of  their  use  for  selection,  training, 

and  career  development  in  a military  setting. 


KAY  H.  SMITH,  Ph.D. 

Chief,  Fort  Benning  Field  Unit 

U.S.  Army  Research  Institute  for  Behavioral  & Social  Sciences 


BACKGROUND 


In  late  1972  an  Assessment  Center  was  established  at  the  directorate 
level  within  the  US  Armv  Infantry  School  (USAIS)  at  Fort  Benning, 
Georgia.  Initial  planning  for  the  Center  was  a joint  effort  of  the 
Office  of  the  Special  Assistant  for  Training  (OSAT)  under  the  Army 
Chief  of  Staff,  and  the  US  Army  Research  Institute  for  Behavioral 
and  Social  Sciences  (ARI),  a class  II  activity  under  the  Office  of 
the  Chief  of  Research  and  Development  (OCRD) . At  the  time  that  the 
Assessment  Center  was  established  at  the  USAIS,  ARI  set  up  a field 
unit  at  Fort  Benning.  R6D  support  for  the  Center  was  its  initial 
primary  mission. 

The  details  of  development  and  operation  of  the  Assessment  Center 
are  more  completely  given  in  the  paper  prepared  for  this  conference 
by  COL  Wallace  F.  Veaudry,  the  Center  Director.  I will  confine  my 
comments  on  the  Center  itself  to  conceptual  issues.  The  Center 
was  tasked  with  two  missions:  1)  Prediction  of  success  in  Officer 

Candidate  School  (OCS) , and  later  success  as  an  officer,  and  2) 
Career  development,  through  assessment  followed  by  performance 
feedback  counseling,  for  Infantry  Officer  Advanced  Course  (lOAC) 
students  and  for  Advanced  Noncommissioned  Officer  Education  System 
(ANCOES)  students.  During  Center  development  the  task  was  broad- 
ened to  include  assessment  of  a sample  of  Infantry  Officer  Basic 
Course  (lOBC)  students  and  a sample  of  ROTC  cadets. 

ASSUMPTIONS  UNDERLYING 
THE  ASSESSMENT  CENTER  CONCEPT 

The  Assessment  Center  concept  is  not  particularly  revolut ionary, 
nor  is  it  new.  The  series  of  performance  requirements  placed  upon 
candidates  for  knighthood  in  medieval  times  might  be  categorized 
as  an  Assessment  Center.  In  modern  times  the  Office  of  Strategic 
Services'  World  War  II  selection  process  described  by  Murray,  e t . a 1 . 
in  The  Assessment  of  Men  stands  as  the  prototype  for  many  current 
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efforts.  In  recent  years  considerable  use  of  Assessment  Centers 
has  been  made  by  members  of  the  industrial  community,  notably 
AT&T,  J.C.  Penney  Co.,  IBM  and  General  Electric.  A number  of 
basic  assumptions  seem  to  underly  the  use  of  Assessment  Centers. 
Among  the  more  important  assumptions  are  the  following; 

1.  The  characteristics  to  be  measured  can  be  identified, 
elicited  at  the  Center,  and  reliably  observed. 

2.  The  characteristics  to  be  measured  require  some  sort  of 
performance  or  situational  testing,  and  that  easily  portable, 
paper  and  pencil  tests  alone  cannot  do  the  job. 

3.  The  charac ter i St i cs  to  be  measured  require  interaction  with 
people,  either  other  examinees  or  role-playing  assessors. 

A.  Situational  "Job  samples"  will  provide  indications  of 
examinee  work  behaviors  which  are  more  standardized  and  controlled 
than  those  shown  in  his  real-life  job,  and  thus  will  make  compari- 
sons between  examinees  more  meaningful. 

5.  Observations  by  trained  assessors  will  be  more  reliable 
and  valid  than  those  made  by  supervisors. 

It  '5  entirely  possible  that  in  a given  application  of  the  assess- 
ment center  concept  some  or  all  of  these  assumptions  may  be  invalid 
Since  centers  are  usually  required  to  measure  such  c'iusive  charac- 
teristics as  communication  skills,  personal  impact,  sensitivity, 
leadership,  etc.,  it  may  be  difficult  to  meet  assumption  number  1. 
If  this  assumption  is  not  met,  of  course,  there  can  be  no  justifi- 
cation for  use  of  Assessment  Centers. 

Measurement  of  reliability  and  validity  become  then  essentially 
the  process  of  proving  that  the  assumptions  are  true. 

VALIDITY  OF  THE  ASSESSMENT  CENTER 
AS  A PREDICTOR  FOR  SELECTION  PURPOSES 

As  mentioned  above,  one  of  the  Center  tasks  is  to  provide  measure- 
ments which  could  serve  as  selection  criteria  for  OCS.  The  basic 
problem  here  seems  relatively  simple,  since  the  criterion  to  be 

predicted successful  completion  of  OCS is  concrete  and  readily 

available.  The  task  is  complicated  somewhat  by  the  hope  expressed 
by  some  high-ranking  Army  officers  that  the  Assessment  Center 
would  allow  the  Army  to  weed  out  men  who  now  successfully  complete 
OCS  but  later  show  flaws  as  officers.  Without  going  into  the  prob- 
lems of  measuring  such  things  as  "character",  these  hopes  suggest 
that  successful  performance  in  OCS  is  not  to  be  regarded  as  an 
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ultimate  criterion.  The  validation  problem,  so  far  as  OC.S  cadets 
are  concerned,  seems  to  be  threefold;  1)  Prediction  of  level 
of  performance  in  OCS,  2)  Prediction  of  quality  of  performance 
as  a commissioned  officer,  3)  Evaluation  of  OCS  training  proce- 
dures and  performance  criteria.  Let's  consider  each  of  these 
separately. 

Prediction  of  performance  in  PCS.  Currently  success  in  OCS 
is  dependent  on  performance  in  three  general  areas:  1)  tradi- 

tional paper  and  pencil  tests  covering  academic  areas,  2)  ratings 
of  leadership  skills  by  peers  and  tactical  officers,  3)  persistence 
in  an  intensive  and  highly  demanding  task  requiring  total  involve- 
ment over  several  weeks'  time.  The  latter  is  reflected  in  the 
measurements  listed  as  1 and  2 but  also  is  shown  by  voluntary 
withdrawal  from  the  program. 

While  the  Assessment  Center  utilizes  a battery  of  standard  paper 
and  pencil  tests  which  will  probably  predict  performance  in  aca- 
demic areas  quite  well,  it  is  obvious  that  such  testing  does  not 
require  an  Assessment  Center. 

It  is  in  the  areas  of  leadership  skills  and  task  persistence  that 
the  Center  shows  the  greatest  promise  for  accounting  for  unique 
variance.  The  situational  tasks  of  the  Center  include  opportuni- 
ties for  observation  of  emergent  leadership  as  well  as  for  evaluation 
of  performance  in  assigned  leadership  tasks.  In  each  of  these  si*:ua- 
tions  multiple  ratings  are  made  by  trained  assessors  and  peer 
evaluations  are  also  obtained.  These  should  provide  good  prediction 
of  the  similar  ratings  obtained  later  in  OCS. 

Since  the  Center  is  a very  demanding  and  totally  involving  experi- 
ence covering  a three  and  one-half  day  period,  it  also  should 
provide  an  opportunity  for  observing  task  persistence  and  dedica- 
tion under  carefully  standardized  conditions.  Coming,  as  it  does, 
in  close  proximity  to  the  beginning  of  OCS  it  should  provide  a 
unique  opportunity  for  prediction  of  this  elusive  factor. 

The  problem  of  predicting  performance  in  OCS  is  a fairly  straight 
forward  one.  A major  difficulty  which  must  be  faced  is  that  our 
limited  sample  size  will  probably  preclude  empirical  predictor 
weighting  with  cross  validation.  Hopefully,  the  Center  will  be 
maintained  beyond  FY74  in  order  to  obtain  further  validation  data. 

Prediction  of  quality  of  performance  as  an  officer.  Valida- 
tion of  Assessment  Center  scores  against  criteria  of  successful 
officer  performance  involves  all  the  problems  that  these  criteria 
have  demonstrated  in  the  past.  Officer  Evaluation  Reports  (OERs) 
are  inflated,  are  not  executed  consistently  by  different  raters, 
and  show  very  little  discrimination.  Used  alone,  OERS  are 
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inadequate  ab  indicators  of  officer  potential.  Graduates  of 
DCS  qo  to  ividely  varying  ass  i gnrienl  s , both  by  branch  of  service 
and  assignments  within  branches.  (The  Army  OCS  is  currently 
branch  i minaler i a 1 . Graduates  qo  on  to  the  officer  basic  course 
in  the  branch  to  which  they  are  assigned.) 

Data  will  be  available  on  performance  in  branch  officer  basic 
courses;  howevci , the  curricula  and  evaluation  procedures  vary 
and  the  numbers  assigned  to  individual  branches  will  be  quite 
sma 1 1 . 

The  major  source  of  data  on  officer  performance  will  have  to  come 
from  Specially  designed  peer,  sui)erior,  and  possibly  subordinate 
ratings  made  after  a specified  t’oriod  of  time  in  the  man's  first 
officer  duly  assignment.  The  variability  contributed  by  difference 
in  assignments  may  be  partially  accounted  for,  by  using  procedures 
for  grouping  of  Junior  officer  assignments  previously  developed 
by  ARI . ' 

One  of  the  problems  to  be  solved  will  be  the  degree  tc;  which  the 
ratings  can  be  made  to  reflect  performance  on  specific  dimensions 
of  behavior.  Research  on  this  problem  is  currently  underway  in  a 
Joint  ARl-HumRRO  effort. 

Evaluation  of  OCS  Training  Procedures  and  Performance  Criteria. 

Ttie  third  area  of  use  of  Assessment  Center  data  follows  naturally 
from  the  two  Just  discussed.  Since  we  will  have  three  sources 
of  performance  evaluation  data,  it  is  logical  to  compare  them. 

The  question  being  asked  is  tiiis.  When  Assessment  Center  data 
and  OCS  performance  scores  made  different  predictions  about  oer- 
formance  of  an  officer  assignment  , which  has  the  greater  probability 
of  being  correct?  This,  of  course,  assumes  that  there  will  be 
discrepancies.  The  more  basic  question  is  this.  Are  the  charac- 
teristics required  for  successful  performance  at  OCS  the  same  as 
those  required  for  successful  performance  in  an  officer  assignment? 

To  the  extent  that  they  are  not,  the  next  question  becomes;  Can 
we  account  for  the  differences  with  Assessment  Center  data? 

To  some  extent,  it  is  hoped  that  some  light  can  be  cast  on  these 
areas  of  concern.  Small  sample  size  will  again  make  definitive 
answers  difficult  to  achieve.  An  area  of  real  concern  involves 
attrition  from  OCS.  This  further  reduces  the  sample  insofar  as 
the  officer  assignment  pertorniance  criterion  is  conccned.  It 
will  provide  positive  validation  data  in  the  case  of  those  with 
poor  Assessment  Center  scores.  The  question  with  regard  to  those 
vjho  do  not  complete  OCS,  but  vvho  have  good  Assessment  Center  scores 
becomes  nioot . If  they  fail  in  OCS  they  become  negative  validation 
cases,  but  if  they  voluntarily  withdraw  the  questions  about  how 
they  might  have  performed  in  OCS  and  later  in  officer  assignments 
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remain  unanswered.  Perhaps  at  least  we  may  find  some  data  which 
will  shed  light  on  why  they  vvithdraw.  Sample  size  will  again  be 
a severe  restriction. 

VAl IDITY  OF  ASSESSMENT  AND  COUNSELING 
PROCEDURES  FOR  CAREER  DEVELOPMENT 


While  the  problems  discussed  in  the  previous  section  are  difficult 
they  are  at  least  familiar.  The  criterion  problem  is  always  with 
us  whenever  performance  is  to  be  predicted.  The  validation  prob- 
lems now  to  be  addressed  include  all  those  problems  but  are  further 
compi icated  by  still  others.  It  is  still  necessary  to  demonstrate 
that  the  behaviors  measured  at  the  Center  relate  to  significant 
dimensions  of  job  performance,  but  now  we  must  also  show  that  going 
through  the  assessment  process  and  receiving  feedback  and  counsel- 
ing on  his  performance  produces  measurable  effects  in  a nan's  later 
performance  and/or  his  subsequent  career  choices. 

While  in  the  previous  section  the  population  to  which  we  were  re- 
ferring involved  DCS  cadets,  this  section  will  have  as  its  main 
focus  NCOs  from  ANCOES  classes  and  captains  from  I OAC . To  a lesser 
extent  .he  discussion  will  also  apply  to  2nd  lieutenants  from  lOBC. 
The  scope  of  this  paper  will  not  permit  detailed  discussion  of  the 
problems  and  procedures  unique  to  each  of  these  population. 

Validation  of  the  measurement  dimensions.  Procedures  for  demon- 
strating the  validity  of  the  Center  measurement  process  will  be  much 
the  same  for  the  counseled  groups  as  for  the  DCS  cadets.  Assessment 
Center  scores  will  be  compared  with  performance  in  the  courses  at 
USAIS,  and  later  with  ratings  of  performance  in  duty  assignments. 
This  process  will  take  some  time  with  the  lOAC  captains  since  they 
are  assessed  at  the  beginning  of  a 39“week  career  course.  The 
other  courses  are  much  shorter,  ranging  around  13  weeks. 

The  major  complication  with  this  validation  of  the  basic  measure- 
ment scales  Involves  the  feedback  counseling.  It  is  assumed  that, 
in  some  cases  at  least,  learning  about  weaknesses  will  lead  to  ac- 
tivity designed  to  strengthen  that  area.  To  the  extent  that  this 
Is  true  ccurel at  ions  between  Assessment  Center  scores  and  later 
criteria  should  be  reduced.  Unfortunately,  constraints  on  time 
and  availability  of  subjects  did  not  allow  a design  involving  an 
assessed  but  not  counseled  subsample  within  the  same  population  as 
an  assessed  and  counseled  sample.  We  may  be  able  to  get  a partial 
answer  to  this  by  comparing  validity  coefficients  of  OCS  cadets 
and  lOBC  2nd  lieutenants  for  officer  assignment  performance  scores. 
This  will  be  tenuous  at  best  since  there  are  several  significant 
differences  between  these  groups  which  could  account  for  any  ob- 
served differences  in  the  validity  coefficients. 
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of  Cuunt.elinq  on  Kitfr  performance-  Si  ..i'  do  noi 
liave  an  assessed  ! it  not  counseled  qroup  what  vve  .;ii!  be  mcasui  inq 
here  is  the  effect  of  the  total  assessment  and  counseling  expel i* 
ence.  The  first  ((uestion  vjill  Ite;  Do  counseled  students  do  better 
at  USAIS  than  tr.. „unseled  students?  Since  the  Assessment  Center 
sample  is  a random  selection  from  the  USAIS  class,  it  should  be 
possible  to  compare  their  school  performance  scores  vjith  the  re- 
mainder of  the  class.  The  kinds  of  school  performance  measures 
available  vary  t rom  population  to  population.  The  lOAC  and  ANCOES 
scores  focus  on  academic  type  tests  while  the  lOBC  classes  produce 
in  addition  to  academics  some  relatively  good  leadership  scores 
from  peer  and  tactical  officer  ratings.  The  constraints  of  the 
school  setting  are  such  that  it  is  unlikely  that  we  will  be  able 
to  obtain  specially  constructed  performance  measures. 

As  with  the  OCS  students,  ratings  will  be  obtained  on  their  per- 
formance in  post-school  duty  assignments.  For  these  purposes  a 
carefully  selected  control  group  will  be  chosen  from  each  class 
•where  students  were  assessed.  These  people  will  be  followed  up 
in  the  same  fashion  as  those  assessed.  In  the  case  cf  the  d.jty 
performance  ratings  it  will  be  possible  to  design  the  measures  in 
such  a viay  as  to  focus  on  particular  weaknesses  which  were  noted 
such  as  reading  and  writing  skills  among  the  NCOs.  These  will 
have  been  a focus  in  counseling  sessions  and  the  performance  ratings 
should  fiiovide  data  on  whether  the  assessees  actual Iv  did  anything 
t)  att'nipt  to  correct  deficiencies. 

Effects  of  counseling  on  career  decisions.  This  will  be  the 
I -t  difficult  area  in  which  to  get  anything  approaching  defini- 
■ ve  answtrrs.  It  lay  be  possible  to  get  information  concerning 
lequested  assignments  and  then  to  compare  these  with  Assessment 
Center  profiles.  It  may  also  be  possible  to  get  comparison  figures 
for  control  and  counseled  groups  on  such  things  as  leaving  the  Army. 
This  will  he  particularly  interesting  for  lOBC  students,  but  will 
require  waiting  3 years  in  most  cases.  The  greatest  bulk  of  data 
in  this  area  will  probably  have  to  come  from  questionnaires  cover- 
ing career  plans.  One  purpose  of  such  questionnaires  vjould  be 
to  assess  the  extent  to  vyhich  there  is  a residual  of  the  profound 
feelings  of  self-understanding  and  new  insight  typically  expressed 
at  the  time  of  the  counseling. 

OTHER  RESEARCH 


The  above  discussion  outlines  in  very  general  terms  some  of  the 
definite  research  plans  v;hich  have  been  dtvelopcd  for  validation 
of  the  USAIS  Assessment  Center.  Some  of  these  are  not  cast  in 
concrete  at  present  and  in  a moment  I will  be  very  happy  to  hear 
comments  and  new  ideas  from  the  floor.  Before  I do  this  hoi'vever 


I would  like  to  mention  briefly  some  of  the  other  research  ques- 
tions we  are  addressing. 

On  the  assessment  process  we  will  be  looking  at  the  following 
areas:  1)  Development  of  a standard  training  program  for  assessors, 

2)  Adaptation  of  assessment  situations  for  training  proposes,  and 

3)  Determining  the  dimensionality  of  our  measurement  space. 

On  the  counseling  process  our  research  goals  are  two-fold:  1) 

Experimental  comparison  of  alternative  feedback  methods  in  order 
to  discover  the  optimal  method,  2)  Development  of  a standard  coun- 
selor training  program. 

Finally  we  will  be  looking  at  profiles  of  scores  for  our  different 
assessment  populations.  These  will  provide  valuable  input  to 
USAIS  regarding  areas  which  should  receive  emphasis  in  training 
courses.  We  also  are  accumulating  very  interesting  normative  data 
on  the  commercial  paper  and  pencil  tests  in  use  at  the  Center. 

No^'  we  get  to  the  main  purpose  of  my  being  here.  Please  feel  free 
to  make  any  comments,  suggestions  or  criticisms  regarding  the  re- 
search problem  of  validation.  Be  as  creative  and/or  fanciful  as 
you  like. 


Patrick  J.  Dowd 


Environmental  Sciences  Division 
USAF  School  of  Aerospace  Medicine 
Aerospace  Medical  Division  (AFSC) 
Brooks  Air  Force  Base,  Texas  7823) 
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1.  INTRODUCTION 


Witliin  man,  a higiiiy  sophisticated  inertial  guidance  system  has  evolved 
phylogenet ically . However,  his  system  is  rather  limited  in  that  it  does 
not  provide  enough  information,  and  at  times  its  vestibular  organs  elicit 
convincing  erroneous  information,  when  man  is  flying  an  aircraft.  It  is  the 
duty  of  flight  instructors  and  trainers  of  student  pilots  to  compensate  for 
these  deficiencies  within  man  so  they  can  avoid  the  sometimes  fatal  con- 
sequences of  spatial  disorientation. 

There  is  increasing  agreement  that  the  present  spatial  orieiftation/ 
disorientation  training  is  not  meeting  the  needs  of  the  pilots.  Therefore, 
those  of  us  interested  in  the  complex  area  of  flight  safety  and  training 
must  not  allow  ourselves  to  be  bound  by  what  exists;  we  must  make  our  own 
recommendations  in  terms  of  what  should  be. 

Historically,  spatial  disorientation  has  been  given  little  or  no  con- 
sideration in  training  plans — in  affecting  adequate  exposure  to  illusions 
or  other  activities  aimed  at  informing  tlie  pilot  of  the  relationship  spatial 
disorientation  has  to  fliglit,  to  iiis  person,  etc. 

in  his  natural  environment  man,  by  using  his  functioning  sense  organs 
and  muscular  effectors,  is  an  almost  perfect  cybernetic  mechanism  for  main- 
taining his  position,  attitude,  or  motion.  In  flight,  however,  man  is  far 
from  perfect  in  this  respect,  especially  when  iie  is  deprived  of  good  visual 
contact  with  the  eartli.  The  degree  of  his  imperfection  depends,  in  part, 
upon  ills  ability  to  respond  to  instruments  which  present  him  abstracted  and 
reduced  information  regarding  these  parameters  and  upon  Ills  skill  in 
interpreting  messages  from  nis  vestibular  and  proprioceptive  systems. 


Maintaining  proper  orientation  in  a high  performance  aircralt  in  weatlier 
is  one  ot  the  most  complex,  nonverbal,  intellectual  functions  that  a man 
can  perform.  it  is  one  that  is  extremely  difficult  to  study  in  situ  because 
of  the  speed  and  distance  involved. 

Laboratory  studies  and  adequate  on-the-ground  training  require  abstrac- 
tion of  the  variables  involved  and  working  with  combinations  of  these 
abstractions,  culminating  with  an  ultimate  integration  of  all  variables  in 
the  aircraft  itself.  A great  many  efforts  in  the  past  have  failed  because 
of  inadequate  definition  of  these  variables,  undue  emphasis  on  the  vestibular 
system  as  it  is  knovm,  reliance  on  face  validity  as  a criterion  in  the 
absence  of  better  data,  etc. 

This  paper  suggests  how  a spatial  orientation  training  device  (1,  2)  can 
be  used  in  conjunction  with  lecture  and  slide  material  to  meet  a threefold 
purpose:  (1)  The  student  will  learn  to  anticipate  the  kinds  of  perceptual 

experiences  he  will  receive  in  tlie  air;  (2)  the  conditions  of  and  mechanisms 
whereby  these  experiences  are  mediated  will  be  elucidated  for  him;  and  (3)  he 
will  practice  maintaining  control  of  the  device  and  performing  desired  maneu- 
vers under  conditions  which  will  disturb  his  orientation  systems  and  lead  to 
erroneous  information. 

A brief  review  of  the  causes  and  costs  of  spatial  disorientation,  the 
present  status  of  spatial  orientation  indoctrination,  and  the  research 
studies  on  the  USAF  School  of  Aerospace  Medicine  spatial  orientation 
trainer  (SOT)  will  be  discussed.  Also,  the  proposed  training  concept  of 
using  a SOT  in  flight  training  for  spatial  orientation  indoctrination  will 


be  presented. 


IL.  CAUSES  OF  SPATIAL  DISORIENTATION 


Man  has  Inherited  and  developed  complex  sensory  systems  which  enable 
iilm  to  move  purposefully  over  the  surface  of  the  earth.  However  well 
adapted  these  systems  are  to  terrestrial  activity,  they  do  not  always 
function  well  in  aircraft.  Let  us  review  the  three  main  sensory  systems 
briefly. 

Vision:  This  is  man's  dominant  sense.  A pilot  flying  on  a clear, 

sunny  day  and  enjoying  a well  defined  horizon  has  little  or  no  difficulty 
v^;ith  spatial  disorientation.  Under  other  conditions,  however,  vision  can 
contribute  to  disorientation.  A tilted  cloud  bank  may  be  accepted  as  a 
true  horizon.  In  the  absence  of  a well-defined  horizon,  ground  lights  may 
become  confused  with  stars,  or  a well-ligtited  stioreline  may  be  accepted  as 
tile  horizon.  Pilots  have  been  known  to  confuse  a star  or  distant  ground 
light  with  wing  iiglits.  TTie  lights  of  other  aircraft  do  not  allow  for  mucii 
depth  perception  or  judgment  of  rate  of  closure,  leading  to  midair  collisions. 
When  a pilot  is  viewing  a single  stationary  light,  he  might  project  imagined 
motion  to  it. 

Proprioception  and  tactile  sets:  Man's  sense  organs  provide  information 

concerning  the  activity  of  skeletal  muscular  groups  and  of  tiie  relative  posi- 
tion and  movements  of  his  limbs.  Tlie  senses  provide  the  recovery  feedback 
for  skilled  movements.  However,  the  various  subsystems  are  subject  to  changed 
loading  by  acceleration;  further,  their  activity  is  modulated  by  vestibular 
inputs.  Tactile  and  deep  pressure  senses  inform  the  individual  of  contact 
with  other  bodies,  that  is,  tlie  eartli,  seats,  control  sticks,  etc.  In  air- 
craft, tile  results  of  such  stimulation  can  be  i rroneous.  Being  pressed 
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against  the  seat  batk.  during  an  accelerated  takeoff  roll  can  give  tlie  pilot 
the  impression  that  the  seat  and  aircraft  are  tilted  backwards;  pressure 
against  the  seat  and  shoulder  belts  in  a down  draft  can  make  him  feel  tliat 
he  is  inverted. 

Vestibular  System:  1 he  vestibule  of  the  inner  ear  serves  in  stabilizing 

the  head  during  locomotion.  One  set  of  organs — the  otolith  organs — senses 
the  attitudes  of  the  head  relative  to  gravity  and  is  an  analog  of  the  artifi- 
cial horizon  on  the  instrument  panel.  Tlie  otlier  set  of  organs — the  semi- 
circular canals — senses  angular  accelerations  imposed  upon  the  head  in  any 
plane  during  walking  or  running.  Otolith  organs  and  semicircular  canals 
send  coordinating  information  to  the  skeletal  muscles,  via  the  brain,  in 
such  a way  that  the  skull  provides  a stable  platform  for  the  eyes  in  walking, 
running,  or  even  more  vigorous  activity. 

When  the  head  is  turned  or  tilted  deliberately,  the  vestibule  assumes 
a different  function;  it  sends  information  of  this  movement  to  the  brain 
stem  to  provide  for  a compensatory  movement  of  the  eyes  in  their  sockets 
so  that  fixation  on  the  viewed  object  is  not  disrupted.  For  this  kind 
of  motion  the  semicircular  canals  signal  rate  of  turn  in  the  appropriate 
plane.  If  the  motion  is  of  short  duration,  the  response  is  accurate. 

However,  due  to  the  mechanical  features  of  the  canal  system,  which  inte- 
grates accelerations  of  sliort  duration  in  order  to  generate  the  transient 
rate-of-turn  signal,  the  system  does  not  register  turns  of  long  duration. 

In  this  case,  it  confuses  completion  of  a turn  in  'one  direction  with  tlie 
initiation  of  a turn  in  the  opposite  sense.  Another  very  serious  error 
generated  by  the  canal  system  is  the  response  to  Coriolis  stimulation; 
if  an  individual  moves  his  iiead  while  riding  turning  equipment,  liis  canals 
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generate  a complex  response,  Che  dominant  feature  of  which  is  a rate-of- 
turn  signal  in  a plane  other  than  that  of  his  turn  or  that  of  his  head 
movement . 


ill.  COST  OF  S1>AT1/\L  DISORIENTATION 

Spatial  d Lsoi ientation  accidents  continue  to  concern  all  military, 
civilian,  aid  commercial  airline  pilots.  A U.S.  Navy  report  listed  395 
naval  aircraft  related  to  spatial  disorientation  accidents  over  a period 
of  8 years,  1958  to  1965  (3).  According  to  Smiley  (1967)  the  primary 
"cause  ot  real  concern  for  safety  in  aircraft"  was  spatial  disorientation 
for  22a  Canadian  military  pilots  (4).  In  1971,  a conference  was  held  by 
the  NAIO  AC.ARi)  Aerospace  Medical  Panel  Meeting  in  Luchon,  France,  on  "The 
Disor iintat  ion  Incident."  Ihis  AGARD  report  (5)  highlighted  the  present 
seriousness  i)f  spatial  disorientation  in  aviation. 

Spatial  disorientation  is  costly  in  lives,  material,  and  money,  as 
illustrated  in  fable  1.  Figures  for  the  USAF  losses  in  1971  represent  only 
those  thu.s  f.ir  identifieil;  to  some  extent,  data  for  1958-1967  may  be  low 
because  oi  the  d i I t i cu 1 1 i i s ol  accident  investigations  in  Vietnam.  Spatial 
disorientation  is  involved  in  a disproportionate  number  of  fatal  accidents. 

Air  Force  data  show  that  overall  safety  has  improved  significantly; 
1955-1958  and  1958-1969  records  show  that  the  annual  rate  of  all  accidents 
was  reduced  by  127  and  of  fatal  accidents  by  82%,  yet  the  annual  rate  of 
spatial  disorientation  accidents  decreased  by  onlv  16%  and  fatal  disorienta- 
tion accidents  by  only  47%.  Comparison  of  i969-1971  data  with  that  of  1958- 
1969  sliows  that  tlie  annual  rate  of  all  accidents  was  reduced  by  10%,  but 
there  was  an  84  increase  for  fatal  accidents;  however,  the  annual  rate  of 
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spatial  disorientation  accidents  decreased  by  50%,  and  of  fatal  disorienta- 
tion accidents  by  59%.  Tlie  increase  in  complexity  and  power  of  the  aircraft 
has  probably  increased  the  dollar  cost  of  spatial  disorientation  even  though 
the  rate  of  loss  lias  diminished  slightly.  This  table  shows  that  spatial 
disorientation  is  a problem  to  other  air  services  as  well.  Except  for  two 
fixed-wing  aircraft,  the  U.S.  Army  spatial  disorientation  accidents  refer  to 
rotary  wing  aircraft.  In  reviewing  other  studies  on  spatial  disorientation 
problems,  Tyler  and  Farr  (14)  reported  that  96%  of  2000  USN  pilots  reported 
one  or  more  disorientation  experiences,  38%  were  hazardous;  Clark  (17)  reported 
that  93%  of  336  pilots  of  all  U.S.  Services  reported  one  or  more  disorientation 
experiences;  Smiley  (4)  reported  that  82%  of  244  RCAF  pilots  reported  expe- 
riencing vertigo  or  disorientation,  8.5%  of  144  pilots  experienced  one  within 
a 1-month  period,  and  6.7%  of  244  pilots  experienced  hazardous  disorientation 
witliin  a 1-year  period;  Doppelt  (12)  reported  that  7.5%  of  55  USAF  pilots 
reported  disorientation  episodes  experienced  within  1000  flying  hours;  and 
recently  Rayman  (18)  listed  a total  of  26  USAF  accidents  resulting  from 
spatial  disorientation  for  the  period  of  January  1966  to  November  1971.  Out 
of  those  26  accidents,  16  resulted  in  fatalities. 

Recent  progress  in  cockpit  and  instrument  design  has  greatly  assisted 
in  lowering  some  ot  the  potential  disorientation  hazards  during  instrument 
flight.  New  and  more  easily  read  instruments  are  constantly  being  introduced. 
Reduction  in  the  number  of  instruments  for  "cross  checking"  to  ascertain 
aircraft  control  has  also  enhanced  the  safety  factor  in  instrument  flying. 

The  relocation  of  radio  and  aircraft  controls  minimized  the  potentially 
hazardous  head  movements  required  during  critical  phases  of  instrument 
flights.  All  tills  progress,  however,  appears  to  be  somewhat  negated  by 
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the  increasing  performance  capabilities  of  today's  aircraft.  To  see  a result 
of  technical  advancement,  one  only  needs  to  examine  the  various  stresses 
a pilot  luidergoes  in  Llie  ever  increasing  accelerations  of  the  USAF  F-15 
aircraft,  or  even  in  some  ii)mmercial  aircraft  sucli  as  the  747  on  takeoff. 

Spatial  disorientation  is  the  most  frequently  reported  biomedical  incident 
threatening  aircraft  crews;  it  is  implicated  in  10%  of  Air  Force  fatal  aircraft 
accidents  recent Iv  studied,  19by-1971  (6).  Tliis  has  been  made  a human-factor 
problem  almost  by  default,  since  flight  instrumentation  supposedly  capable 
of  providing  tiie  pilot  with  sufficient  information  for  completely  safe  IFR 

operations  has  proven  to  be  prohibitively  expensive  in  initial  cost  and  in 

( 

maintenance,  and  also  of  low  reliability. 

iV.  PRESENT  STATUS  OF  SPATIAL  ORIENTATION  iNDOCTRlNATlON 

Within  the  Air  Force,  the  problem  confronting  flight  surgeons,  phys- 
iological training  officers  (PTO) , and  others  interested  in  eliminating  spatial 
disorientation  accidents  is  threefold:  first,  to  demonstrate  effectively  to 

pilots  tliat  spatial  disorientation  can  happen  to  anyone;  second,  to  discern 
tliroug.n  tilling  wliich  pilots  might  be  prone  to  lose  aircraft  control  under 
disorienting  maneuvers;  and,  third,  to  offer  pilots  a way  to  become  resistant 
to  the  elli'cts  ui  disorienting  influences.  The  SOT  was  designed  to  study  the 
problem  of  disorient.it ion  from  these  three  aspects. 

Current  ,\i r Force  practice  is  to  expose  the  pilot  once  a year  to 
indocl r inat i on  lilms  and  lectures,  and  then  to  fly  him  through  a set  series 
I'f  maneuvers  under  the  hood,  returning  control  of  the  aircraft  to  him  at 
certain  points  in  the  maneuvers.  The  maneuvers  listed  in  the  flight  train- 
ing syllabus  are;  liidiing  while  turning  (posturogravic-  or  somatogravic- 
type  illusion),  diving  during  turn  recovery  (posturogravi o-type  Illusion), 
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tilting  to  right  or  left  (leans),  reversal  of  motion  (similar  to  a spin), 
diving  or  rolling  beyond  vertical  (Coriolis  or  G-excess  effect),  and 
sensation  oi  climbing  (posturogravic-type  illusion) . 


r 


Some  bases  add  the  demonstration  of  motion-induced  illusions  by  using  a 
Bdr^ny  chair.  Either  each  pilot  rides  the  chair  and  experiences  the  false 
sensations  of  turning  and/or  the  Coriolis  illusion,  or  one  pilot  is  used  to 
demonstrate  his  reactions  and  relate  to  the  observing  pilots  his  false  sensa- 
tions of  turning  or  rolling  or  pitching  while  being  rotated  about  the  vertical 
axis  and  tilting  his  head  in  either  the  sagittal  or  lateral  planes. 

The  undergraduate  pilot  training  (UPT)  instructor  staff  and  students  at 
Laredo  AFB  (T-37  and  T-38  phases  of  training)  reported  on  their  spatial 
orientation/disorientation  indoctrination  program  as  follows: 

a.  Lectures  and  films  on  spatial  disorientation  are  presented 
(2  to  3 hours  on  films,  2 hours  on  lectures). 

b.  All  students  participate  in  a demonstration  on  the  Bdrany 
chair  and  experience  the  Coriolis  illusion. 

c.  The  best  and  most  frequently  used  film  is  the  Canadian  Air 
Force  "The  Perception  of  Orientation,"  and  the  next  best  is  the  USAF  film 
"Pilot  Vertigo." 

d.  Two  hours  of  flying  unusual  attitudes  are  imposed  on  the 
students  by  the  UPT  instructors  in  flight  demonstrations  occurring  halfway 
in  the  T-37  phase  (instrumentation  phase)  and  again  at  the  beginning  of 
the  T-38  phase. 

e.  Most  commonly  experienced  illusions  in  flights  are  listed  as: 

(1)  leans;  (2)  Coriolis;  and  (3)  posturogravic  (oculogravic  or  somatogravic) . 

f.  Most  of  the  students  seemed  pleased  with  the  present  orienta- 
tion training  program. 
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g.  Most,  students  felt  that  motion  in  relation  to  attitude  could 
be  learned  and  that  some  spatial  orientation  trainer  should  be  available  for 
such  a program,  particularly  before  instrument  training. 

l’h'’s  i o logi  ca ! training  officers  (PTOs)  expressed  a need  for  a more 
realistic  training  program  in  whicli  a simulator  with  a capacity  of  fly-out 
controls  is  provided  and  a need  for  more  research  with  "operational  orienta- 
tion." Muny  of  them  also  expressed  concern  over  unfamiliarity  with  content 
material  involved  in  spatial  orientation  indoctrination.  By  content  material 
they  meant  tlie  physiological  contributing  factors  involved  in  spatial  orienta- 
tion and  di SOI ieiitatioa,  and  the  technological  knowledge  of  aerodynamics  in 
motion  stimuli.  Since  they  had  never  experienced  the  illusions  and  were 
unable  to  describe  them,  some  PTOs  felt  that  tiiey  were  not  trained  sufficiently 
to  instruct  oi  indoctrinate  pilots  or  student  pilots  on  the  problems  of  spatial 
disorientation. 

Two  1 light-trained  PTOs  were  concerned  about  the  inadequacy  of  spatial 
orientation  training  in  that  career  aircrewmen  were  not  able  to  use  any  ground- 
based  demonstrators  such  as  a Bdrany  cliair  and  that  the  topic  of  spatial  dis- 
orientntion  Was  iioL  inc.luded  in  their  physiological  training  refresher  course 
taken  every  tnree  years.  uue  of  these  PTOs  even  claimed  that  his  own  flight 
instructor  wa.i  "not  that  familiar  witli  what  he  was  trying  to  demonstrate"  when 
he  undertook  in-flight  demonstrations  of  unusual  attitude  maneuvers. 

Lt  is  apparent  from  L'nese  interviews  that  tlie  present  requirements  for 
spatial  orientatitjn  indoctrination  are  not  adequate  nor  standardized  in 
1 1 ight  tr. lining. 
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V.  RESEARCH  STUDIES  ON  THE  SPATIAL  ORIENTATION  TRfUNER  (SOT) 


Studies  have  been  conducted  to  determine  the  SOT's  effectiveness  in 
producing  sensations  and  perceptions  similar  to  those  imposed  by  real  air- 
craft during  various  disorienting  flight  maneuvers  and  to  estimate  its  use- 
fulness and  acceptability  in  a flight  training  program.  From  the  interviews 
of  more  than  600  USAF  pilots  concerning  their  experiences  with  flight  spatial 
disorientation,  the  common  flight  motion-induced  illusions  in  relating  attitude 
to  motion  in  an  aircraft  were:  leans,  posturogravic  (oculogravic  or  "jet  upset"), 

and  Coriolis  illusions.  The  SOT  is  capable  of  simulating  ail  of  these  commonly 
experienced  illusions,  and  the  procedure  of  demonstrating  these  illusions  in  the 
SOT  is  described  elsewhere  (19-22) . 

Twenty-two  student  pilots  in  the  instrument  phase  of  flight  training 
who  were  exposed  to  various  disorienting  maneuvers  in  the  SOT  reported  that 
the  SOT  (a)  increased  their  confidence  in  instrument  flying  as  they  were 
able  to  cope  witti  the  takeoff  illusion  (a  common  posturogravic  illusion)  in 
a real  takeoff,  (b)  increased  their  faith  in  the  Instruments  as  reporters 
of  real-world  conditions  and  in  their  ability  to  respond  to  them,  (c)  helped 
them  to  maintain  the  desired  attitude  by  instruments  in  spite  of  the  illusions 
experienced  (an  important  condition  for  a man-in-the-loop  trainer)  , (d) 
closed  the  credibility  gap  by  convincing  the  pilots  that  their  sensations 
can  be  at  dangerous  variance  with  the  true  facts,  (e)  familiarized  them  with 
certain  disorienting  maneuvers  that  they  had  not  yet  experienced  in  their 
in-flight  unusual-attitude-maneuvers  training  phase,  and  (f)  afforded  tliem  the 
unrestricted  opportunity  to  work  out  solutions  and  make  corrections  by  instru- 
ments to  motion-induced  orientation  difficulties  that  they  were  not  able  to 
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do  in  actual  tliglit  lor  fear  of  violating  the  aircraft  structural  limitations 
or  tlie  allotment  of  air  space,  fuel,  time,  or  altitude  (20). 

Student  pilots,  who  are  not  expected  to  be  fully  conversant  with  the 
bodily  sensations  induced  by  flight  which  are  familiar  to  pilots  with  long 
histories  of  instrumented  flight,  learned  to  perform  successfully.  They 
respondod  ti'  different  SOT  induced  illusions  by  relating  these  illusions  to 
those  iliey  experienced  in  flight.  This  indicates  a valuable  point  for  such 
a tra.ner--t..,  students  were  capable  of  vivid  perceptual  responses  to  motion 
becausi  iiu-re  i-  a great  deal  of  commonality  between  tlie  SOT  and  the  aircraft 
as  tai  ,1-.  • . lec.tat  i.ui  meclianisins  are  concerned  (20,  22). 

Ill)  a t ,,u  h less  qualified  than  student  pilots  liave  been  able  to  oper- 

ate the  sol'  on  relevant  illusions  in  a simulated  situation  confounded  by  other 
illusii)ns.  c'ramer  and  Wolfe's  study  (23)  showed  that  USAF  basic  trainees 
expel"  iei'ced  tin-  appropriate  perceptual  responses  to  changes  in  pitch  motion 
and  accur.itil-  solved  an  orientation  problem;  they  adjusted  pitch  setting 
without  biiug  diverted  by  tlie  irrelevant  illusion  of  roll.  A later  study  (24) 
contirwed  tlie  .lijility  of  such  nonflying  personnel  to  correct  for  pitch  changes 
even  under  alcoholic  aftereffects.  Botii  studies  stressed  an  important  point 
in  simulator  training — that  man  is  capable  of  learning  attitude  changes  in 
re  1 a t i on  to  mo  1 1 i;n . 

A dilferent  study  by  Wolfe  and  Cramer  (25)  illustrated  an  important 
concept  in  tiiat  even  sophisticated  pilots  were  shown  to  be  highly  susceptible 
to  illusions  ol  , itch  during  accelerated  takeoff  or  deceleration  (posturogravic 
or  somatogravic  or  Jet  upset) . This  stud'’  shewed  that  the  experiences  high- 
performance  aircraft  pilots  must  reintegrate  otolithic  sensations  with  visual 
information  from  their  instrumentation  to  "cancel  out"  such  disorienting  illu- 
sions of  pi  tell. 
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In  a study  assessing  the  SOT  (21) , comparisons  were  made  of  other  selected 


orientation  training  devices  in  relation  to  U.S.  Air  Force  flight  training  in 
the  T-38  airplane.  Assessment  categories  were:  motion  parameters  (yaw,  roll, 

pitch,  and  Gx  axis  acceleration) ; illusions  that  each  device  can  generate 
(graveyard  spin,  Coriolis,  leans,  posturogravic , jet  upset,  inversion,  elevator 
effect,  autokinesis , oculogyration , false  horizontal  cues  of  clouds  and  lights); 
type  of  instrumentation  available  to  the  pilot-subject  in  relation  to  attitude 
and  control;  reaction  capability  provided  in  each  device  for  the  subject  to 
recover  or  correct  his  attitude;  reproducible  capability  to  measure  and  eval- 
uate the  subject's  performance  in  coping  with  disorienting  maneuvers;  invul- 
nerability to  weather;  safety;  and  initial  and  operating  costs.  The  compar- 
isons indicated  that  the  SOT  far  excelled  other  ground-based  devices  for  both 
hardware-  and  pilot-oriented  assessment  criteria.  This  study  emphasized  the 
similarity  and  reproducibility  of  SOT-induced  Illusions  to  those  experienced 
in  flight  and  the  positive  feedback  to  the  subject  in  his  selfrecovery  by  using 
instruments.  The  training  concept  of  the  SOT  was  stated  in  terms  of  transfer 
of  learning — that  is,  that  senses  can  be  trained  to  relate  attitude  to  motion 
and,  through  practice  on  a SOT,  sensory  conflict  can  be  minimized  by  instrument 
feedback. 

Finally,  an  evaluation  study  was  made  with  six  pilot  instructors  in  under- 
graduate pilot  training.  All  six  pilot  instructors  recommended  this  type  of 
trainer  to  be  used  before  a student  enters  the  flight  instrument  phase  of 
training.  All  six  instructors  liked  the  disorientation  "symptoms"  induced  by 
the  SOT. 

Table  II  shows  approval  of  the  SOT  by  various  Air  Force  groups  who  had 
been  exposed  to  its  use  as  an  aid  to  spatial  orientation  training. 
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VI.  OPERATIONAL  EXAMPLE  OF  THE  SOT 


When  a pilot  takes  oil  in  a high  performance  aircraft,  he  experiences 
an  exaggerated  sensation  of  being  pitched  upward — a common  distortion  of 
vertical ity  as  perceived  by  the  pilot.  If  tie  reacts  to  this  sensation  with- 
out properly  scanning  his  instruments,  he  is  apt  to  put  his  craft  into  a full 
power  crasli.  Ihis  postiirogravic  or  soraatogravic  illusion  can  readily  be 
acliieved  in  the  SOT  by  having  the  pilot  establish  his  perceptual  straight  and 
Level  condition  without  using  his  instruments  on  tlie  SOT  panel.  To  illustrate 
tills  effect,  tile  SOT  cockpit  is  turned  inward  and  is  rotated  about  the  main 
yaw  axis  to  produce  a centripetal  acceleration  of  approximately  1 g.  When 
told  by  tlie  instructor  to  make  the  proper  adjustments  for  straight  and  level, 
the  pilot  automatically  pitches  the  cockpit  to  a nose-down  position — as  mucii 
us  20  degrees.  Theii,  the  instructor  informs  liim  of  his  error  by  allowing  him 
to  read  the  uttituue  indicator.  The  pilot  continues  to  practice  this  maneuver 
and  eventually  learns  where  straight  and  level  fligiit  is.  Thus,  in  the  train- 
ing period,  the  pilot  learns  to  associate  different  sets  of  perceptual  criteria 
with  different  conditions,  in  this  case,  witli  resultant  linear  acceleration 
vectors  of  different  magnitudes.  Tliough  this  may  be  oversimplifying  a dif- 
ficult and  damp  lex  learning  t.isk,  it  is  one  way  of  demonstrating  man's  capacity 
to  cope  with  such  motion  stimuli.  It  is  just  a beginning. 

Operational  research  regarding  other  flight  functions  in  relation  to  the 
SOT  is  essential.  Tiic  real  justification  for  using  a SOT  in  USAF  flight  train- 
ing is  to  minimise  spatial  disorientation  accidents  and  incidents. 

Vll.  A PROPOSED  SPATIM.  ORIENTATION  TR.\1N1NG  CONCEPT 

At  present  there  is  no  effective  way  to  practice  spatial  disorientation 
maneuvers  in  the  undergraduate  pilot  training,  either  in  flight  or  in  a ground 
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environment.  Indoctrination  on  the  physiological  factors  and  tlieir  relation- 
ship to  physical  stimuli  that  induce  spatial  disorientation  is  undoubtedly 
essential  for  training  student  pilots,  pilots,  and  navigators.  A more  effec- 
tive indoctrination  program  in  spatial  orientation  is  necessary  to  familiarize 
and  train  the  pilot  (1)  in  identifying  and  recognizing  certain  unusual  attitude 
maneuvers,  and  (2)  in  convincing  the  pilot  to  check  his  instruments  when  his 
sensory  information  may  be  in  conflict  or  in  error  with  the  actual  flight  maneuver. 

Devices  developed  for  research  in  spatial  disorientation  at  the  USAF 
School  of  Aerospace  Medicine  are  illustrated  in  Figure  1.  The  SOT  is  the  present 
climax  of  this  effort.  To  my  knowledge  the  SOT  is  the  only  available  closed- 
loop  device  developed  solely  to  simulate  motion-induced  spatial-disorienting 
maneuvers  and  provide  motion  cues  needed  by  the  pilot  if  he  is  to  successfully 
develop  spatial  orientation  skills. 

The  block  diagram  in  Figure  2 is  necessarily  incomplete.  It  is  presented 

I 

solely  to  demonstrate:  (1)  the  pervasiveness  of  the  effects  of  stimuli  on  the 

various  senses  involved  in  orientation;  (2)  the  complexity  of  the  mechanisms 
serving  Instrumented  flight;  and  (3)  the  sensory  systems'  input  to  the  overall 
central  nervous  system's  perception  of  motion  to  attitude. 

In  this  diagram  the  central  nervous  system  (CNS)  can  take  advantage  of 
instrument  information  in  order  to  correct  or  adjust  the  sensory  information 
input.  This  continuous  flow  of  effects  and  information  between  sensory  systems 
and  environmental  stimuli  ultimately  accounts  for  successful  spatial  orienta- 
tion, even  in  an  environment  of  intense  stressful  modifications  (false  visual 
cues,  accelerations,  gravity,  etc.).  Both  instrument  and  sensory  feedback, 
and  the  capacity  to  respond  to  it  are  essential. 

The  diagram  in  Figure  3 attempts  to  present  the  systems'  interfaces  in 
spatial  orientation  in  both  the  aircraft  and  a SOT-type  simulator.  It  is 
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hypothesized  that  positive  transfer  of  training  will  occur  from  a SOT  to  an 
airplane  since  the  learning  processes  of  relating  motion  cues  to  attitude 
is  similar  in  both. 

In  analyzing  the  pilot's  spatial  orientation  processes,  we  find  our- 
selves confronted  with  a series  of  major  questions.  What  influences,  internal 
or  external,  act  upon  man's  perceptual  system  and  how  are  such  influences  com- 
municated? As  depicted  here,  all  this  is  still  at  a very  crude  level  of  for- 
mulation. It  will  be  the  task  of  future  researchers  in  flight  training  and  in 
spatial  orientation  training  to  refine  these  relationships. 

Figure  A Illustrates  the  methods  and  aids  presently  used  in  spatial 
orientation  indoctrination,  as  described  earlier  in  this  report.  Using  a 
SOT -type  simulator  as  a training  device  for  sensory  adaptation  and  learning 
to  relate  attitude  to  motion  cues,  as  well  as  demonstrating  and  familiarizing 
the  student  pilot  with  the  common  motion-  and  visually-induced  illusions  in 
flight,  is  the  innovative  approach  recommended  here. 

Studies  of  spatial  disorientation  are  overwhelmingly  of  the  type  in  which 
the  subject  is  a passive  transducer  of  a relatively  simple  vestibular,  pro- 
prioceptive, or  visual  stimuli  (26-32).  There  are  very  few  experiments  in 
which  the  subject  is  part  of  a closed-loop;  i.e.,  his  responses  exert  some 
influence  upon  the  stimuli  to  which  he  is  responding,  which  is  the  situation 
of  a pilot  trying  to  maintain  proper  spatial  orientation  in  his  aircraft 
(33-36) . Even  these  few  studies  are  somewhat  limited  in  that  they  are 
essentially  one-dimensional;  i.e.,  activity  is  confined  to  one  axis  alone. 

They  say  nothing,  for  Instance,  of  judgment  of  pitch  or  roll  attitude  con- 
founded by  illusions  of  pitch  or  roll  Induced  by  linear  accelerations  as  in 
takeoff,  or  by  Coriolis  accelerations  induced  by  noise  abatement  turns  imme- 
diately after  takeoff,  or  by  combinations  of  both  of  these  illusions. 


In  addition,  these  studies  provide  no  information  of  the  susceptibility  of 
the  orientation  mechanisms  to  training.  The  SOT  provides  students  not  only 
an  adequate  visual  input  by  way  of  his  instruments,  but  gives  them  pro- 
prioceptive and  vestibular  stimulations  to  simulate  flight  (19).  Since  the 
appropriate  stimuli  for  the  vestibular  and  proprioceptive  systems  are  posi- 
tion and  its  successive  derivatives  with  respect  to  time,  viz.,  velocity  and 
acceleration,  it  is  manifestly  impossible  to  reproduce  exactly  the  stimuli 
generated  by  an  airplane.  As  a result,  a considerable  amount  of  abstraction 
from  the  real  flight  situation  must  be  performed.  One  must  know  how  a partic- 
ular orientation  problem  "looks"  to  the  vestibule.  For  example,  how  can  the 
same  flight  orientation  problem  be  presented  to  the  vestibule  in  ground-based 
operations?  How  closely  does  the  SOT  approximate  a flight  illusion  in  order 
to  inform  the  student  pilot  what  to  expect  in  actual  flight  and  how  it  differs 
from  what  is  presented  to  him  in  the  SOT? 

In  an  emergency  situation,  man  generally  reacts  in  accordance  with  his 
earliest  similar  experience.  This  fundamental  rule  of  learning  explains  why 
it  is  safer  to  teach  the  use  of  aircraft  instruments  right  from  the  start. 

Researchers  agree  that  learning  transference  can  occur  (37-39) . Since 
learning  is  considered  a developmental  process,  prior  learning  plays  an 
important  role  in  transfer:  new  learning  tasks  are  placed  within  the  context 

of  habits  already  established  from  previous  experience.  Thus,  a student  pilot 
learning  a new  situation  relies  on  the  bank  of  appropriate  habits  already 
mastered.  Practice  in  the  SOT  will  help  the  pilot  control  the  aircraft  in 
accordance  with  what  he  learns  from  his  flight  instruments,  rather  than  hand- 
icap him  by  a generalized  nonselective  switching-off  of  his  sensory  system's 
input.  Presumably,  through  practice  on  motion  simulating  devices,  proper 
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processing  within  the  central  nervous  system  of  combined  inputs  from  various  < 

sensory  modalities  should  reduce  tlie  number  of  conditions  which  different 
sensory  receptors  may  be  signaling,  and  assist  the  pilot  in  correctly  iden- 
tifying and  responding  to  those  conditions  imposed  upon  him  by  flight  (23-25, 

40,  41,  44). 

A basic  concept  in  learning  and  in  perception  is  that  man  uses  all  sensory 
modalities  to  orient  himself  in  space;  lie  does  not  ignore  any  of  them.  The 

I 

pilot's  central  nervous  system  seeks  as  much  information  as  possible  from  the  i 

senses  in  order  to  accurately  perceive  his  own  spatial  orientation  and  that 
of  his  craft  (23-25) . External  instrument  cues  help  the  pilot  correct  or 
interpret  some  sensory  information  that  may  conflict  with  his  or  his  craft's 
true  orientation  (41,  44). 

Profound  changes  occur  in  the  pilot  as  he  learns  to  fly  by  instruments. 

He  becomes  more  proficient  in  decoding  information  from  his  instruments  as  he 
practices  either  in  aircraft  or  in  static  flight  simulators.  The  fate  of  the 

input  from  his  vestibule  is  still  a matter  of  debate.  Some  argue  that  the  I 

pilot  learns  to  inhibit  his  vestibular  meclianisms  so  that  the  input  itself  is 
attenuated.  A more  realistic  theory,  however,  is  that  reduction  of  vestibular 
responses  occurs  with  repeated  stimulation  as  a result  of  central  changes 

since  even  the  peripheral  inhibitory  system  is  subject  to  central  control  i 

(23-25).  There  is  also  a possibility  that  the  pilot  learns  to  more  correctly  ; 

interpret  the  information  that  he  receives  from  his  vestibule,  so  that  he 

operates  the  controls  with  good  integration  of  information  from  his  instru-  j 

ments  and  from  his  vestibular-kinesthetic  system. 

This  philosophy  of  distrusting  motion  cues  and  suppressing  vestibular 
and  proprioceptive  sensations  is  basically  against  the  generally  well-accepted 
rule  that  learning,  retention,  and  motivation  are  all  improved  by  Involving 
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as  many  o£  the  trainee's  sense  modalities  as  possible  (40,  41).  No  pilot 
ignores  his  sensations,  but  he  uses  tliem  to  draw  his  attention  to  his  instru- 
ments to  determine  wliere  liis  aircraft  is  (40-44). 

In  our  studies  of  student  pilots,  pilot  instructors,  and  advanced  jet 
pilots,  we  found  tliat  ttiey  c.in  readily  learn  to  recognize  and  identify  var- 
ious aspects  of  tlieir  sensations  of  position  and  its  derivations  (20,  23-25). 
Miiny  of  the  pilots  relate  certain  SOT  motions  to  flight  maneuvers  in  their 
commentaries  while  operating  the  SOT  controls.  Pilots  also  state  that 
visual  displays  of  lights  sucii  as  ground  lights,  and  star  lights,  and  cockpit 
and  instrument-display  lights,  are  needed  to  induce  some  oi  tlie  visual 
illusions  that  they  encounter  in  niglit  flying.  Tliey  also  mention  tliat  tlie 
cockpit  in  the  SOT  should  be  as  similar  as  possible  in  configuration  to  an 
aircraft,  preferably  the  T-38. 

Pilots  claim  that  one  of  their  problems  is  to  successfully  perform 
within  a disorienting  maneuver.  Tliey  need  more  realistic  training  of 
unusual  attitude  maneuvers  on  a motion  system  ground-based  device  that  will 
(1)  simulate  as  accurately  as  possible  the  positional  disorienting  sensa- 
tions they  experience  in  flight,  and  (2)  have  instruments  to  indicate  what 
tliey  are  recovering  from,  or  by  which  they  can  correct  an  attitude.  In 
flight,  pilots  do  not  have  the  time  nor  the  capability,  regardless  of  the 
safety  problem,  to  practice  such  recovery  techniques. 

Thorough  research  is  recommended  of  a spatial  orientation  training 
program,  providing  two  to  three  hours  of  instruction  in  spatial  orientation, 
for  use  before  the  instrument  flight  training  phase  of  the  UPT  program. 

This  program  should  include  a dialogue  between  Instructor  and  the  student 
regarding  the  physiological  systems  that  mediate  illusions.  Then  these 


r—  1 

physiologically  induced  illusions  should  be  presented  within  the  context 
of  flying  in  the  SOT,  so  that  tlie  student  can  recognize  them  and  interpret 
them  for  wliat  tliey  are.  Using  his  instruments  the  student  will  operate  the 
SOT  controls  and  then  will  be  reinforced  or  criticized  upon  his  performance 
in  recovering  from  or  in  coping  with  disorienting  maneuvers  within  the  SOT. 

In  summary,  the  main  training  concepts  of  using  the  SOT  for  spatial 
orientation  in  flight  training  are:  (1)  tlie  basic  training  function  is  to 

convince  the  pilot  that  liis  sensations  cm  vary  dangerously  from  the  true 
facts  — attacking  the  credibility  gap;  (2)  vestibular  habituation  or 
suppression  results  from  repeated  vestibular  stimulation  in  that  vestibular 
and  proprioceptive  sensory  inputs  from  motion  are  suppressed,  but  this  is 
a rather  tenous  suppression;  (3)  vestibular  and  proprioceptive  senses  can 
be  formative  if  used  correctly  through  training  and  reinforcement  on  a SOT- 
type  simulator. 

VIII.  CONCLUSIONS  j 

Studies  using  the  SOT  indicate  that  pilots  use  tlieir  vestibular  and 
proprioceptive  senses  as  formative  instruments  rather  than  ignore  or  suppress 
these  senses  (23-25,  35,  41),  which  is  the  natural  way  for  man  to  perceive 
and  correct  for  any  sensory  conflict  (41)  . The  pilot  can  correct  sensory 
conflict  by  identifying  the  problem  through  familiarization  or  experience, 
and  by  cross-checking  external  cues  such  as  attitude,  motion,  and  directional 
instriuiients  to  help  him  orient  himself  (41,  44).  Resistance  to  disturbing  and 
disorienting  maneuvers  is  also  increased  by  familiarization  and  practice  with 
such  stinuli  (2B,  34,  35). 

The  SOT  has  been  valuable  in  showing  the  need  for  a closed-loop  motion 
trainer  to  iielp  a pilot  learn  to  recover  from  and  cope  with  spatial  disorienting 
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maneuvers.  The  staff  at  USAFSAM  who  have  investigated  the  problem  of  spatial 
orientation/disorientation  training  for  the  past  decade  have  found  that  the 
trainee  must  receive  positive  feedback  of  the  true  position  of  the  trainer 
and  what  illusion  the  trainer  is  simulating. 

The  validity  of  any  training  program  for  spatial  orientation  will  be 
established  by  comparing  functions  obtained  in  the  trainer  with  those  from 
the  same  population  in  the  aircraft  (41)  . Based  on  design  studies  and  cross- 
validation  of  learning  transference,  a safe  and  economical  means  of  providing 
pretraining  spatial  disorientation  maneuvers  and  corrective  tasks  is  feasible. 
The  transf er-or-training  potential  of  this  SOT,  as  illustrated  in  Figures  3 
and  4,  should  significantly  reduce  the  actual  flight  time  required  for  the 
pilot  to  reach  desired  levels  of  in-flight  proficiency  or  to  cope  with  a 
spatial  disorienting  maneuver. 

Along  with  using  a SOT  within  a spatial  orientation  training  syllabus, 
we  recommend  supplementary  material  on  the  senses  involved,  keyed  to  demon- 
stration and  training  within  a SOT-type  simulator.  Together,  they  can  help 
the  student  pilot  overcome  sensory  conflicts  that  result  in  disorientation 
and/or  practice  proper  aircraft  control  in  spite  of  disorienting  sensory 
cues . 
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Figure  1.  USAFSAM  Ground-Based  Vestibular  Stimulators. 
Figure  2.  Servomechanisms  in  Spatial  Orientation. 
Figure  3.  Systems  Interfaces  in  Spatial  Orientation, 
Figure  A.  Spatial  Disorientation  Indoctrination. 
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• TRUE  INDICATION  OF  POSITION  Of  COCKPIT  IN  RELATION  TO  EARTH 


* CLOSED-LOOP  MOTION  SIMULATOR 


MINUTES 


1973  MTA  Steen' no  Committee  Meetinq 

The  1973  Military  Testing  Association  (MTA)  Steerino  Committee  Meeting 
convened  at  San  Antonio,  Texas  at  1330,  30  October  1973.  The  following 
members  were  present; 
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Cdr  M.  W.  Faes«^el 
Mr.  C.  J.  Macaluso 

U.  S.  Arrny 

Col  W.  F.  Veaudry 
Dr.  E.  R.  Ousek 
Mr.  W.  W.  Wance 
Mr.  Anthony  Strigari 


U.  S.  Marine  Corps 

Maj  H.  E.  Wells 
Mr  Edward  A.  Dover 

U.  S.  Coast  Guard 

Cdr  K.  R.  Depperman 
Mr.  John  A.  Burt 
Cdr  K.  M.  Roughgarden 

Austral ia 

Lt  Cdr  J.  E.  C.  Williams,  RAM 
Sqn  Ldr  W.  J.  Watson,  RAAF 

Canada 

Cdr  W.  H.  Northey,  CF  Hg 


The  meetina  was  called  to  order  by  the  MTA  President  who  presented  the 
following  Agenda  items  to  be  discussed:  By-Laws,  the  1974  MTA  meeting, 

organizational  name  change,  and  miscellaneous  items  requiring  committee 
act’on. 

1.  Mr.  Jaimes,  AFHRL,  opened  the  disucssion  regarding  the  funding  of 
MTA  meetings  outlining  the  AFR  requirements  as  ennumerated  in  AFR  80-43. 
Discussion  was  invited  regarding  the  fee  versus  a registration  fee  for 
futiding  purposes.  Under  AFR  80-43,  an  Air  Force  ornanization  hosti ng 

a meeting  or  symposium  is  expected  to  pay  registration  fees  for  aTl  UoD 
participants.  A coordi nator  on  the  other  hand  pays  no  such  fees.  The 
present  meeting  is  being  funned  by  a very  small  amount  of  AF  funds  plus 
a registration  fee  imposed  on  each  conferee.  The  registration  fee  this 
year  will  include  the  cost  of  printing  the  MTA  Proceedings.  A copy  of 
the  proceedings  will  be  forwarded  to  each  attendee. 

2.  Mr.  John  Burt,  USCG,  made  a iix)tion  to  change  the  term  "host"  in  the 
current  By-Laws  to  "coordinating."  Cdr  Roughgarden,  USCR,  seconded  the 
motion  which  carried  with  no  opposition. 
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3.  Dr.  Brokaw,  AFHRL,  moved  that  the  name  of  the  MTA  should  be  changed. 

This  motion  was  seconded  by  Mr.  Anthony  Strigari,  USA,  and  the  final  vote 
was  5 to  4 in  favor  of  the  proposed  action.  A further  motion  was  made 

by  Dr.  Brokaw  concerning  the  method  to  be  used  in  effecting  a name  change. 
Dr.  Brokaw  will  circulate  correspondence  to  Steering  Committee  members 
seeking  suggestions  for  name  changes.  When  they  are  returned,  if  there  is 
a consensus,  he  will  prepare  a ballot  which  will  be  distributed  to  the 
membership,  defined  as  registrants  to  this  year's  conference,  for  their 
completion.  Ballots  will  then  be  returned  to  the  Coast  Guard  in  Oklahoma 
City,  and  results  will  be  reported  at  next  year's  MTA  meeting.  The  motion 
was  seconded  by  fir.  Strigari,  USA,  and  passed  unanimously. 

4.  Upon  a query  by  the  U.  S.  Coast  Guard  concerning  the  cost  of  the  MTA, 

Mr.  Jaimes,  AFURL,  stated  that  he  will  prepare  a financial  statement,  and 

a transfer  of  remaining  funds  will  be  made  to  the  Coast  Guard.  In  addition, 
a transfer  of  the  fTTST  tape  files  used  in  mail-out  will  be  made  available 
to  the  Coast  Guard. 

b.  Dr.  brokaw,  in  order  to  simplify  procedures  for  the  prooram  chairman 
of  the  74  conference,  offers  the  following  sungestions  to  the  Coast  Guard; 

a.  In  your  call  for  the  papers,  provide  a form  to  be  completed  some- 
what like  that  of  the  American  Psychological  Association  or  of  the  Air 
Force  Academy.  On  this  form,  the  would-be  participant  must  indicate  the 
overall  time  for  his  paper  plus  discussion,  he  must  acknowledge  awareness 
of  a requirement  for  photo  reproducible  cooy  to  be  used  in  the  proceedings, 
and  careful  specification  of  that  copy  should  be  made.  For  example,  single 
spaced  with  1 1/4  inch  margins  on  either  side,  1 inch  on  top  and  3/4  inch 
on  the  bottom,  8 by  10  1/2  paper.  It  must  be  taped  with  either  a carbon 
tape  or  with  a ribbon  designed  for  photo  offset  reproduction,  etc, 

b.  The  Air  Force  Academy  uses  a procedure  which  could  be  very  useful 
in  terms  ot  expediting  the  publication  of  the  proceedings  and  also  the 
clearance  of  papers.  The  Air  Force  Academy  requires  a reproducible  copy 
be  delivered  to  them  not  less  than  30  days  prior  to  the  opening  day  of 
their  conference  with  indication  to  the  participant  that  his  paper  will 
be  withheld  if  clearance  is  not  forthcoming. 

6.  We  had  logistic  and  financial  problems  which  prevented  our  doing  so, 
but  there  was  considerable  evidence  that  we  might  have  had  an  even 
larger  turnout  if  we  had  been  able  to  furnish  a tentative  program  to 
our  mai  1 inq  list. 

7.  It  is  probably  inevitable  that  you  will  have  a few  last  minute 
cancellations  and  it  has  been  suagested  that  various  means  might  be 
taken  to  counter  such  changes.  For  example,  persons  in  the  local  area 
who  have  a paper  ready  for  presentation  and  who  are  amenable  to  "filling 
in"  could  be  used.  We  had  about  three  of  those.  However,  I see  no  way 
to  avoid  the  somewhat  unfair  treatment  of  those  persons  who  would  have 
participated  except  their  papers  had  been  rejected. 
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8.  Commander  Northey,  CDF,  suggested  that  the  concluding  remarks  by  the 
MTA  President  solicit  comments  from  the  MTA  participants  as  to  whether 
the  single-session,  one-room,  or  a multiple-track  meeting  with  two  or 
three  sessions  operating  simultaneously  was  preferred  for  the  next  meetinq. 

9.  Colonel  F.  L.  McLanathan  (Retired)  moved  that  a citation  be  awarded 
to  an  outstanding  MTA  member  on  an  annual  basis  in  the  name  of  Captain 
Harry  Greer,  1).  S.  Navy  (Deceased)  and  presented  at  the  I'TTA  banquet. 
Commander  Northey,  CDF,  seconded  the  motion.  The  following  procedure 
for  nomination  was  outlined.  Nominations  for  the  award  will  be  submitted 
to  the  Steering  Committee  for  consideration  and  final  selection.  The 
responsibility  for  the  award  and  accompanying  citation  will  be  the 
responsibility  of  the  coordinating  agency  for  that  meeting.  This  motion 
unanimously  carried. 

Dr.  Brokaw,  MTA  Secretary  (AFHRL),  moved  to  adjourn.  C.  J.  Macaluso, 

USN,  seconded  and  the  meetinq  was  adjourned. 
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MILITARY  TESTING  ASSOCIATION 
HARRY  H.  GREER  AWARD 

In  the  memory  of  Captain  Harry  H.  Greer,  USN,  founder  of  the 
Military  Testing  Association,  the  first  Harry  H,  Greer  award  is  here- 
by presented  to  Charles  J.  Macaluso,  United  States  Naval  Examining 
Center,  for  his  outstanding  services  to  the  organi zation . From  its 
inception,  the  Military  Testing  Association  has  been  indebted  to  this 
man. 

Through  foresight,  devotion,  and  dedication,  he  has  made  a sub- 
stantial contribution  to  raising  the  stature  of  the  association  from  a 
small  discussion  group  to  a major  international  organization  which  now 
provides  a forum  for  specialists  in  many  areas  of  personnel  evaluation. 
Charles  J.  Macaluso  was  instrumental  in  planning  the  first  meeting  of 
the  association  in  1959  and  each  year  since  he  has  been  a major  factor 
in  the  success  of  its  conventions.  He  has  devoted  countless  hours  to 
furthering  the  aims  of  the  association  across  the  Department  of  Defense 
and  in  other  nations.  His  zeal  and  energy  in  developing  the  objectives 
of  the  association  have  been  an  example  and  inspiration  to  all  its 
members.  It  is  with  gratitude  and  appreciation  that  the  Military 
Testing  Association  recognizes  the  efforts  of  Charles  J.  Macaluso  with 
the  Harry  H.  Greer  award. 
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BY-LAWS  OF  THE  MILITARY  TESTING  ASSOCIATION 
Article  I - Name 

The  name  of  this  organization  shall  be  the  Military  Testing  Association. 

Article  II  - Purpose 

The  purpose  of  this  Association  shall  be  to: 

A.  Assemble  representatives  cf  the  various  armed  services  of  the  United 
States  and  such  other  nations  as  might  request  to  discuss  and  exchange  ideas 
concerning  assessment  of  military  personnel. 

B.  Review,  study,  and  discuss  the  mission,  organ’zation,  operations, 
and  research  activities  of  the  various  associated  organizations  engaged  in 
military  personnel  assessment. 

C.  Foster  improved  personnel  assessment  through  exploration  and  pre- 
sentation of  new  techniques  and  procedures  for  behavioral  measurement,  occu- 
pational analysis,  manpower  analysis,  simulation  models,  training  programs, 
selection  methodology,  survey  and  feedback  systems. 

D.  Promote  cooperation  in  the  exchange  of  assessment  procedures,  tech- 
niques and  instruments. 

E.  Stimulate  and  coordinate  research  in  the  various  areas  of  military 
personnel  systems. 

F.  Promote  the  assessment  of  military  personnel  as  a scientific  adjunct 
to  modern  military  personnel  management  within  the  military  and  professional 
communities. 


Article  III  - Participation 

The  following  categories  shall  constitute  membership  within  the  MTA: 
A.  Primary  Membership. 

1.  All  active  duty  military  and  civilian  personnel  permanently 
assigned  to  an  agency  of  the  associated  armed  services  having  primary 
responsibility  for  assessment  for  personnel  systems; 

2.  All  civilian  and  active  duty  military  personnel  permanently 
assigned  to  an  organization  exercising  direct  command  over  an  agency  of 
the  associated  armed  services  holding  primary  responsibility  for  assess- 
ment of  military  personnel. 
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B.  Associate  f^mbership . 


1.  Membership  in  this  category  will  be  extended  to  permanent 
personnel  of  various  governmental,  educational,  business,  industrial  and 
private  organi zations  engaged  in  activities  that  parallel  those  of  the 
primary  membership.  Associate  members  shall  be  entitled  to  all  privileges 
of  primary  members  with  the  exception  of  mentiership  on  the  Steering  Committee. 
This  restriction  may  lie  waived  by  the  majority  vote  of  the  Steering  Committee, 

Article  IV  - Dues 

No  annual  dues  shall  be  levied  against  the  participants. 

Article  V - Steering  Committee 

A.  I he  governing  body  of  the  Association  shall  be  the  Steering  Com- 
mittee. Tfie  Steering  Committee  shall  consist  of  voting  and  non-voting 
members.  Voting  members  are  primary  members  of  the  Steering  Committee. 

Primary  membership  shall  include: 

1.  Ihe  Commanding  Officers  of  the  respective  agencies  of  the 
armed  services  exercising  responsibility  for  personnel  assessment  programs. 

2.  The  ranking  civilian  professional  employees  of  the  respective 
agencies  of  the  armed  service  exercising  primary  responsi bi 1 i ty  for  the 
conduct  of  personnel  assessment  systems.  Each  agency  shall  have  no  more 
than  two  (2)  professional  civilian  representatives. 

B.  Associate  membership  of  the  Steering  Committee  shall  be  extended 

by  majority  vote  of  the  committee  to  representatives  of  various  governmental, 
educational,  business,  industrial  and  private  organizations  whose  purposes 
parallel  those  of  the  Association. 

C.  The  Chairman  of  the  Steering  Committee  shall  be  selected  through 
election  by  the  majority  vote  of  the  committee.  The  term  of  office  shall 
be  one  year  and  shall  begin  the  last  day  of  the  annual  conference. 

D.  The  Steering  Committee  shall  have  general  supervision  over  the 
affairs  of  the  Association  and  shall  have  the  responsibility  for  all  activities 
of  the  Association.  The  Steering  Committee  shall  conduct  the  business  of  the 
Association  in  the  interim  between  annual  conferences  of  the  Association  by 
such  means  of  communication  as  deemed  appropriate  by  the  President  or  Chairman. 

E.  Meeting  of  the  Steering  Committee  shall  be  held  during  the  annual 
conferences  of  tiie  Association  and  at  such  times  as  requested  by  the  President 
of  the  Association  or  the  Chairman  of  the  Steering  Committee.  A majority  of 
the  members  of  the  Steering  Committee  shall  constitute  a quorum. 
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Article  VI  - Officers 


A.  The  officers  of  the  association  shall  consist  of  a President, 
Chairman  of  the  Steering  Committee  and  a Secretary, 

B.  The  President  of  the  Association  shall  be  the  Commanding  Officer 
of  the  armed  services  agency  coordinating  the  annual  conference  of  the 
Association.  The  term  of  the  President  shall  begin  at  the  close  of  the 
annual  conference  of  the  Association  and  shall  expire  at  the  close  of  the 
next  annual  conference. 

C.  It  shall  be  the  duty  of  the  President  to  organize  and  coordinate 
the  annual  conference  of  the  Association  held  during  his  term  of  office, 
and  to  perform  the  customary  duties  of  a president. 

D.  The  Secretary  of  the  Association  shall  be  filled  through  appoint- 
ment by  the  President  of  the  Association.  The  term  of  office  of  the 
Secretary  shall  be  the  same  as  that  of  the  President. 

E.  It  shall  be  the  duty  of  the  Secretary  of  the  Association  to  keep 
the  records  of  the  association,  and  the  Steering  Committee,  and  to  conduct 
official  correspondence  of  the  association,  and  to  issue  notices  for  con- 
ferences. The  Secretary  shall  also  perform  such  additional  duties  and 
take  such  additional  responsibilities  as  the  President  may  delegate  to  him. 

Article  VII  - Meetings 

A.  The  Association  shall  hold  a conference  annually. 

B.  The  annual  conference  of  the  Association  shall  be  coordinated  by 
the  agencies  of  the  associated  armed  services  exercising  primary  responsi- 
bility for  military  personnel  assessment  in  order  of  the  following  rotating 
schedule : 

United  States  Army 
United  States  Marine  Corps 
Uni  ted  States  Navy 
United  States  Air  Force 
United  States  Coast  Guard 

C.  The  annual  conference  of  the  Association  shall  be  held  at  a time 
and  place  determined  by  the  coordinating  agency.  The  membership  of  the 
association  shall  be  informed  at  the  annual  conference  of  the  place  at  which 
the  following  annual  conference  will  be  held.  The  coordinating  agency  shall 
inform  the  Steering  Committee  of  the  time  of  the  annual  conference  not  less 
than  six  (6)  months  prior  to  the  conference. 
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D.  Ihe  coordinatinq  agency  shall  exercise  planning  and  supervision 
over  the  program  of  the  annual  conference.  Final  selection  of  program 
content  shall  be  the  responsibility  of  the  coordinating  organization. 

E,  Any  other  organization  desiring  to  coordinate  the  conference 
may  submit  a formal  request  to  the  Chairman  of  the  Steering  Committee,  no 
later  than  18  months  prior  to  the  date  they  wish  to  serve  as  host. 

Article  VIII  - Committees 

A.  Standing  committees  may  be  named  from  time  to  time,  as  required, 
by  vote  of  the  Steering  Committee.  The  chairman  of  each  standing  committee 
shall  be  appointed  by  the  Chairman  of  the  Steering  Committee.  Members  of 
standing  committees  shall  be  appointed  by  the  Chairman  of  the  Steering 
Committee  in  consultation  with  the  chairman  of  the  committee  in  question. 
Chairmen  and  committee  members  shall  serve  in  their  appointed  capacities 
at  the  discretion  of  the  Chairman  of  the  Steering  Committee.  The  Chairman 
of  the  Steering  Comnittee  shall  be  an  ex  officio  member  of  all  standing  corn- 
mi  ttees. 


de'i*-  « i th  the  counsel  and  approval  of  the  Steering  Committee 
nay  appon*  . ■ a*  ho  conni  ttees  as  are  needed  from  time  to  time.  An  ad  hoc 
commit^tee  ~.p  until  its  assigned  task  is  completed  or  for  the  length 

of  ti  ie  . ' ' . tP(  President  in  consultation  with  the  Steering  Committee. 

. - • 'inr’ttt-es  shall  clear  their  general  plans  of  action  and 

new  pn  I-  ••  Steering  Committee,  and  no  committee  or  committee 

chairnidn  -r\i  relationships  or  activities  with  persons  or  groups 

outside  of  • ■ ition  that  extend  beyond  the  approved  general  plan  of 

work  witnou*  ’*•  specific  authorization  of  the  Steering  Committee, 

D.  In  tiie  interest  of  continuity,  if  any  officer  or  member  has  any  duty 
elected  or  appointed  placed  on  him,  and  is  unable  to  perform  the  designated 
duty,  he  should  decline  and  notify  at  once  the  officers  of  the  association 
that  he  cannot  accept  or  continue  said  duty. 

Article  IX  - Amendments 

A.  Ammendments  of  these  By-Laws  may  be  made  at  any  annual  conference 
of  the  Association. 

B.  Amendments  of  the  By-Laws  may  be  made  by  majority  vote  of  the 
assembled  membership  of  the  Association  provided  that  the  proposed  amend- 
ments shall  have  been  approved  by  a majority  vote  of  the  Steering  Committee. 

C.  Proposed  amendments  not  approved  by  a majority  vote  of  the  Steering 
Committee  shall  require  a two-third's  vote  of  the  assembled  membership  of  the 
Association. 


Article  X - Voting 


All  members  in  attendance  shall  be  voting  members. 

Article  XI  - Enactment 

These  By-Laws  shall  be  in  force  immediately  upon  acceptance  by  a 
majority  of  the  assembled  membership  of  the  Association  and/or  amended 
(in  force  2 November  1973). 
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LISTING  OF  CONFEREES 

ABELLERA,  James  W.,  ILt 

AFHRL/PE,  Lackland  AFR,  Texas  78236 
ALDRICH,  Theodore  B.,  Col 

AFHRL/MD,  Lackland  AFB , Texas  78236 
ALEXANDER,  Helen  A. 

AFHRL/MD,  Lackland  AFB,  Texas  78236 
ALLEN,  Eugene  D. , Capt 

Department  of  Education  Evaluation,  CE  School,  AFIT/DEE,  Wright 
Patterson  AFB,  Ohio  45433 
ALLEN,  Marlin  R. , Capt 

AFHRL/SM,  Lackland  AFB,  Texas  78236 
ALLEY,  William  E. 

AFHRL/PE,  Lackland  AFB,  Texas  78236 
ANDERSON,  Charles 

Educational  Testing  Service,  960  Grove  Street,  Evanston,  Illinois  60201 
ANSBRO,  Thomas  M. 

Service  School  Command,  Naval  Training  Center,  Orlando,  Florida  32813 
ARCHER,  Joann  R. 

AFHRL/OR,  Lackland  AFB,  Texas  78236 
ARCHER,  Wayne  B. 

AFHRL/OR,  Lackland  AFB,  Texas  78236 
BADALAMENTE,  Richard  V.,  Maj 

3714  36th  Street,  Lubbock,  Texas  79413 
BAILEY,  James  T.,  Sgt 

AFHRL/SM,  Lackland  AFB,  Texas  78236 
BAKER,  Thomas  6. 

ATSL-TE,  USA  Ordnance  Center  & School,  Aberdeen  Proving  Ground, 

Maryland  21005 
BARTON,  Bruce  B.,  AlC 

AFHRL/SM,  Lackland  AFB,  Texas  78236 
BELT,  Dr.  John  A. 

Wichita  State  University,  Wichita,  Kansas  67208 
BERBERICH,  George  L.,  Amn 

AFHRL/PE,  Lackland  AFB,  Texas  78236 
BERGE,  Paul 

USAFI , 2318  S.  Park  Street,  Madison,  Wisconsin  53713 
BERGMAN,  Joseph 

AFHRL/OR,  Lackland  AFB,  Texas  78236 
BERTHOLD,  Oscar  A,,  Col 

AFHRL/CCV,  Brooks  AFB,  Texas  78235 
BEST,  William  J.,  AlC 

AFHRL/SM,  Lackland  AFB,  Texas  78236 
BIEDIGER,  Rosalie  A. 

AFHRL/SM,  Lackland  AFB,  Texas  78236 
BIRDSALL,  Walter 

Naval  Examining  Center,  Bldg  2711,  Great  Lakes,  Illinois  60088 
BLAIR,  Ben  R. , Jr. 

3700  OCMST,  Lackland  AFB,  Texas  78236 
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BOHATCH,  Andrew  R. 

Naval  Examining  Center  Bldg  2711,  Great  Lakes  Naval  Training  Center, 
Great  Lakes,  Illinois 
BOLLER,  Lawrence  J.,  Maj 

Office  of  Manpower  Utilization,  HQ  MC  AOIH,  Marine  Corps  Base, 
Quantico,  Virginia  22134 
BONDARUCK,  John 

National  Security  Agency,  Department  of  Defense,  Ft  Meade,  Maryland 
20730 

BOTTENBERG,  Dr.  Robert  A. 

AFHRL/SM,  Lackland  AFB,  Texas  78236 
BOWSER,  S.  E. 

Navy  Personnel  Research  & Development  Center,  Point  Loma,  San  Diego, 
California  92152 
BOYD,  Dr.  Joseph  L. 

Educational  Testing  Service,  Princeton,  New  Jersey  08540 
BRAUN,  F.  B. 

Data-Design  Laboratories,  803  W.  Broad  Street,  Falls  Church,  Virginia 
22046 

BRAZEL,  James  L. 

AFHRL/SM,  Lackland  AFB,  Texas  78236 
BROKAW,  Dr.  Lei  and  D. 

AFHRL/PE,  Lackland  AFB,  Texas  78236 
BROWN,  Bruce,  Cdr 

DIET  Staff,  Pensacola,  Florida  32508 
BROWN,  Thomas  A. 

Info  Sciences  & Math  Department,  RAND,  Santa  Monica,  California  90406 
BR'JMBACK,  Dr.  Gary  B. 

AIR,  8555  16th  Street,  Silver  Spring,  Maryland  20910 
BULLOCK,  Kay  D. 

United  Services  Automobile  Association,  4119  Broadway,  San  Antonio, 
Texas  78288 
BURT,  vlohn  A. 

IJ.  S.  Coast  Guard  Institute,  P.  0.  Substation  18,  Oklahoma  City, 
Oklahoma  73169 
BUSBY,  Charles  W.,  Cdr 

U.  S.  Coast  Guard,  USCG  Training  Center,  Petaluma,  California  94952 
CALHOUN,  Gary  J.,  AlC 

AFHRL/SM,  Lackland  AFB,  Texas  78236 
CALIAHAN,  James  P. 
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